Pro User
  • All of Microsoft
  • My Role
    • All in My Role
    • Pro User
    • Administrator
  • Focus
    • All in Focus
    • Microsoft Ignite 2026
    • Microsoft Copilot
    • All about AI
    • M365 Admin
    • What's Hot
    • M365 Release
Timespan
explore our new search
Microsoft Foundry: improve AI agents continuously
Developer

Microsoft Foundry: improve AI agents continuously

Microsoft Foundry links agent monitoring, evaluation and optimization so teams can test improvements before releasing them.

Key insights

  • NTT DATA is applying the improvement approach to a program-management agent intended to scale across more than 1,000 internal projects.
  • The improvement loop uses production traces and feedback to identify recurring problems, then tests possible changes.
  • Teams can assess changes to instructions, skills, tools or model selection against evaluation criteria.
  • Human review and approval remain part of the proposed release process.

Microsoft Foundry brings observability, evaluation and optimization together in a continuous-improvement approach for AI agents. It matters to teams building and operating agents that need to assess quality, safety, cost and performance as conditions change. Microsoft Azure presents the approach through its account of NTT DATA’s work on a program-management agent.

What it does

The approach treats agent improvement as a repeatable operating cycle rather than a one-time test before deployment. Teams establish baselines, review production traces and user feedback for recurring problems, and test potential changes against evaluation criteria. The cycle can then be repeated as the agent and its business context evolve.

NTT DATA is using observability, evaluations and optimization to identify weaknesses in its program-management agent. The stated goal is to expand the agent across more than 1,000 internal projects. The example puts the focus on operational processes for managing agents, not just building a prototype.

Who it is for and what you need

This approach is relevant to organizations developing and operating AI agents, including teams responsible for agent quality and governance. It starts with defined baselines and evaluation criteria, along with production activity that teams can analyze. The evaluation criteria can reflect business requirements such as task completion, accuracy, relevance and safety.

Teams also need a way to review proposed changes before release. The approach combines automated analysis and testing with human review and approval; it is not described as an agent changing itself without oversight.

How it works

  1. Set baselines for the agent’s quality, safety, cost and performance.
  2. Review traces and user feedback to find recurring failures or other issues.
  3. Test possible adjustments to instructions, skills, tools or model selection.
  4. Evaluate candidates against defined criteria, then review the results before choosing a change for release.
  5. Repeat the cycle using new operational evidence.

Traces provide evidence about agent activity, while evaluations help teams judge results against selected criteria. Comparing candidates across quality, latency and cost can help expose trade-offs rather than focusing on answer quality alone.

Limitations and alternatives

Evaluation depends on having criteria that reflect the organization’s needs; a score alone does not decide whether a proposed change should ship. The approach also requires review of candidate changes before release. NTT DATA’s example concerns a program-management agent, so it illustrates the approach in that context rather than establishing results for every agent workload.

For teams that currently rely on manual tuning, the described alternative is a recurring, evidence-based cycle: use production observations to identify issues, test changes, and review results before release.

Sources

Frequently asked questions

How does Microsoft Foundry support continuous improvement for AI agents?

The approach uses production traces and feedback to find recurring issues, then evaluates potential changes to instructions, skills, tools or model selection.

Does the agent make and release its own improvements?

No. The approach combines automated discovery and testing with human review and approval before a change is released.

How is NTT DATA using the approach?

NTT DATA is applying observability, evaluations and optimization to a program-management agent, with plans to expand it across more than 1,000 internal projects.