Pro User
  • All of Microsoft
  • My Role
    • All in My Role
    • Pro User
    • Administrator
  • Focus
    • All in Focus
    • Microsoft Ignite 2026
    • Microsoft Copilot
    • All about AI
    • M365 Admin
    • What's Hot
    • M365 Release
Timespan
explore our new search
Microsoft Foundry: Improve AI agents continuously
All about AI

Microsoft Foundry: Improve AI agents continuously

Microsoft Foundry teams can define success criteria, evaluate agents, and optimize quality, cost, and performance as business needs change.

Key insights

  • Define the business outcome and grading criteria before choosing a model.
  • A rubric evaluator can be generated from an agent's specification and used to grade simulated conversations.
  • The session organizes development into three stages: make it work, make it better, and make it scale.
  • Optimization options include prompt changes, grounding, task decomposition, model routing, post-training, caching, and architecture.
  • The travel-agent example scores task completion, policy compliance, and grounding.

Microsoft Foundry's continuous-improvement approach gives teams a way to define what an AI agent should accomplish, evaluate its results, and optimize the system around measurable business goals. It is aimed at developers and technology teams building agents for business processes, rather than people looking only to select a model. Microsoft Events demonstrates the approach with a travel-agent scenario and a rubric-based evaluation workflow.

What's new

The central idea is to treat an AI application as a system that needs ongoing evaluation and improvement, not as a finished product once it can answer a prompt. An agent may respond correctly yet fail to resolve a user's problem or complete the intended business action. The session's example is a travel portal where employees need trips submitted and approved, compliance managers need policy adherence, and the CFO needs spending kept within budget.

The proposed process has three stages: make it work, make it better, and make it scale. Start by defining the business outcome and what counts as success; then build a working prototype. Next, improve quality, cost, and performance, with people and agents working together. Finally, monitor models in production and use built-in controls to continue improving and governing them.

This puts the business specification ahead of model selection. In the travel example, the stated goal is to make the right policy decision for every request, quickly and at the lowest cost to serve. Those criteria can become an evaluation rubric, so a team can compare agent behavior against its requirements rather than relying on an impression that one model or prompt is better.

Who it is for and what you need

The approach is for developers and technology builders who are implementing AI systems around business outcomes. The demonstration uses Microsoft Foundry and a travel-agent prototype, but the presenters describe the scenario as applicable to other business processes as well. Its practical starting point is a written specification: the outcome the agent should produce and the criteria that define a good result.

The example uses GitHub Copilot connected to the Foundry MCP server to build from the agent specification. The evaluator is then created from the agent's instructions, with proposed dimensions such as task completion, policy compliance, and grounding. Teams should treat those dimensions as a starting point to align with their own process and success criteria.

How it works

The demonstration turns the agent's requirements into a repeatable evaluation rather than testing a few answers manually. A rubric evaluator scores responses across defined dimensions; grounding, in this context, means whether the answer is supported by the relevant information. A combined score can help teams track whether a change improves the result against the rubric.

  1. Write down the business outcome before choosing a model. Specify what a successful task looks like for the user and the organization.
  2. Set grading criteria in plain English. For the travel example, the rubric covers task completion, policy compliance, and grounding, with weights contributing to a score.
  3. Use the specification and agent instructions to create a rubric evaluator in Foundry. Add any additional context needed for the evaluation.
  4. Create an evaluation against the agent and simulate conversations. In the demonstration, the system asks 100 questions and grades each answer against the rubric.
  5. Use the results to guide optimization. The session identifies prompt optimization, content grounding, task decomposition, model routing and post-training, caching, and system architecture as potential levers.

Model choice is one part of that work, not the whole architecture. The session describes Foundry as bringing together models, agent capabilities, knowledge, tools, and safety systems. It also presents the model catalog as a way to evaluate options against consistent quality, safety, and business criteria without rebuilding the AI stack every time a model choice changes.

Limitations and alternatives

A rubric is only useful when its criteria reflect the outcome the organization actually needs. A high score against a poorly chosen rubric could still represent a poor business result, so the specification should be grounded in the intended task and policy. The session's advice is to define the criteria before selecting a model, rather than retrofitting success measures after development.

Improvement also involves trade-offs: the stated goals include quality, cost, and performance, and the session calls out latency as an optimization target. A change that helps one measure may not be the right choice for the overall business outcome. Use the rubric and success criteria to evaluate changes instead of assuming that a newer or larger model is automatically better.

The workflow described is an engineering process for defining, evaluating, and optimizing an agent. It does not mean an agent independently changes its own production behavior without oversight. The session frames scaling as monitoring models and using controls to keep improving and governing them. Teams should therefore make the evaluation criteria and intended business result explicit before they use scores to guide changes.

Sources

Frequently asked questions

How do you evaluate an AI agent in Microsoft Foundry?

Define success criteria in the agent specification, create a rubric evaluator, then evaluate simulated conversations against dimensions such as task completion, policy compliance, and grounding.

What should I define before choosing an AI model?

Define the business outcome and plain-English grading criteria first. These criteria become the rubric for evaluating candidate models and later changes.

What can teams optimize in an AI agent?

The session lists prompt optimization, content grounding, task decomposition, model routing and post-training, caching, and system architecture.