Azure AI Search: KBs in Custom Agents

Key insights

• This you_tube_video demo from the Microsoft 365 & Power Platform community call shows Ayça Baş connecting a custom engine agent to Azure AI Search.
  The presenter demonstrates how an agent can query indexed claim data and return claim-related answers.
• A knowledge base in Azure AI Search acts as an orchestration layer, not as the raw data.
  It tells agents which knowledge sources to query and how to combine results, and ties into Foundry IQ as a reusable knowledge layer.
• Microsoft uses agentic retrieval where an LLM helps with query planning.
  The LLM breaks questions into subqueries, expands terms, and uses context so the search system finds better matches.
• Key building blocks include a knowledge source, a knowledge base, and the runtime retrieve action or agent call.
  You must create at least one source before the base, both on the same service, and management happens via REST APIs and preview SDKs.
• Benefits include improved retrieval quality, parallel searching across multiple sources, and strong reusability for many agents and apps.
  The approach also supports mixed data access and enforces permission-aware grounding to respect user permissions.
• The demo’s practical steps: connect an agent to indexed claim data, create knowledge sources, register search tools, and make the agent answer claim questions with citations.
  These steps show how teams can turn existing indexes and remote sources into a single, reusable retrieval layer.

Introduction

Microsoft 365 published a video demonstration on integrating Azure AI Search with custom engine agents, and this article summarizes the key points for editorial review. The demo was presented by Ayça Baş during a Microsoft 365 & Power Platform community call and focuses on connecting agents to indexed claim data. Consequently, the session highlights both practical steps and architectural concepts that matter for teams building retrieval-augmented applications. Overall, the presentation frames the work as part of Microsoft's broader move toward reusable knowledge layers for AI agents.

Demo overview and objectives

The video walks viewers through creating a knowledge base that sits above one or more knowledge sources and uses Azure AI Search to answer claim-related questions. First, the presenter demonstrates how to register search tools and link indexed claim data as a searchable source, and then she shows how the agent queries that source. Next, the demo explains how to configure the retrieval action so the agent can return synthesized answers with citations. As a result, viewers get a practical example of turning indexed content into an accessible knowledge layer.

How the system works in practice

At the core, Microsoft distinguishes between a knowledge source and a knowledge base: the former points to indexed or remote content and the latter orchestrates retrieval across those sources. During the demo, the knowledge base issues parallel queries, collects results, and optionally uses an LLM to plan, synthesize, or summarize responses. Moreover, the agentic retrieval approach lets the model decompose user questions into subqueries, which improves precision when users use varied terminology. Therefore, the flow becomes planning, parallel search, result merging, and answer synthesis.

Key building blocks and APIs

Microsoft emphasizes that you must create a knowledge source before creating a knowledge base, and both must reside on the same Azure AI Search service in current releases. The demo uses REST APIs and preview SDKs to show creation and retrieval calls, and it highlights the retrieve action as the runtime entry point for agentic searches. Additionally, Microsoft points to integrating both indexed and some remote sources like SharePoint or web content, although availability depends on API version and preview status. Consequently, developers should expect an evolving surface while evaluating integration paths.

Advantages and tradeoffs

This design offers clear benefits: by using an LLM-driven query planner, the system can rewrite or broaden queries, which improves recall and helps when users phrase questions loosely. Furthermore, parallel searching across multiple sources reduces latency and centralizes retrieval logic, making it easier to reuse a single knowledge base across agents and applications. However, tradeoffs exist: relying on LLM planning and multi-source routing increases system complexity and may add cost and operational overhead. Therefore, teams must weigh improved retrieval quality against added integration and maintenance effort.

Challenges and practical considerations

One challenge is ensuring consistent access control and permission-aware grounding when multiple sources with different security models are involved, which Microsoft addresses in part through its Foundry tooling. Another difficulty lies in debugging and tracing how subqueries are planned and routed, since agentic retrieval introduces intermediate steps that can obscure end-to-end behavior. Moreover, organizations should plan for versioning and service co-location requirements, because the knowledge source and base currently need to live on the same search service. Consequently, architects must balance flexibility against operational constraints and compliance needs.

Best practices and recommendations

To get started, teams should index a representative sample of content and then validate query planning using real user questions; this helps reveal where expansion or rewriting helps and where it harms precision. Additionally, it is wise to monitor cost and latency under realistic loads, because parallel queries and LLM calls can increase both. Finally, document governance and permission mappings early so that agents respect data access policies as the knowledge base scales across applications. Thus, a measured rollout with testing, monitoring, and governance reduces risk while proving value.

Conclusion and implications for adopters

Microsoft’s demo provides a useful, hands-on look at building a searchable knowledge layer with Azure AI Search and custom agents, and it makes the case for reusable knowledge bases in agent architectures. While the approach improves retrieval quality and reusability, it also introduces tradeoffs in complexity, cost, and operational demands that teams must manage. Ultimately, organizations that adopt this pattern should plan for iterative testing, clear governance, and close attention to tracing and permissions so they can realize benefits without exposing themselves to avoidable risks. In short, the demo is a helpful blueprint, but practical success depends on careful design and ongoing management.

Keywords

Azure AI Search knowledge base, Custom Engine Agents, Azure Cognitive Search, semantic search Azure, vector search Azure, knowledge management Azure, retrieval augmented generation Azure, enterprise search solutions Azure