Microsoft Foundry brings observability, evaluation and optimization together in a continuous-improvement approach for AI agents. It matters to teams building and operating agents that need to assess quality, safety, cost and performance as conditions change. Microsoft Azure presents the approach through its account of NTT DATA’s work on a program-management agent.
The approach treats agent improvement as a repeatable operating cycle rather than a one-time test before deployment. Teams establish baselines, review production traces and user feedback for recurring problems, and test potential changes against evaluation criteria. The cycle can then be repeated as the agent and its business context evolve.
NTT DATA is using observability, evaluations and optimization to identify weaknesses in its program-management agent. The stated goal is to expand the agent across more than 1,000 internal projects. The example puts the focus on operational processes for managing agents, not just building a prototype.
This approach is relevant to organizations developing and operating AI agents, including teams responsible for agent quality and governance. It starts with defined baselines and evaluation criteria, along with production activity that teams can analyze. The evaluation criteria can reflect business requirements such as task completion, accuracy, relevance and safety.
Teams also need a way to review proposed changes before release. The approach combines automated analysis and testing with human review and approval; it is not described as an agent changing itself without oversight.
Traces provide evidence about agent activity, while evaluations help teams judge results against selected criteria. Comparing candidates across quality, latency and cost can help expose trade-offs rather than focusing on answer quality alone.
Evaluation depends on having criteria that reflect the organization’s needs; a score alone does not decide whether a proposed change should ship. The approach also requires review of candidate changes before release. NTT DATA’s example concerns a program-management agent, so it illustrates the approach in that context rather than establishing results for every agent workload.
For teams that currently rely on manual tuning, the described alternative is a recurring, evidence-based cycle: use production observations to identify issues, test changes, and review results before release.
The approach uses production traces and feedback to find recurring issues, then evaluates potential changes to instructions, skills, tools or model selection.
No. The approach combines automated discovery and testing with human review and approval before a change is released.
NTT DATA is applying observability, evaluations and optimization to a program-management agent, with plans to expand it across more than 1,000 internal projects.