Azure AI infrastructure for models and agents
Compute

Azure AI infrastructure for models and agents

Azure is bringing compute, storage, networking and agent controls together to support AI workloads from model inference to autonomous agents.

Key insights

  • Azure frames AI infrastructure around three workload types: training, inference and agentic AI.
  • Cobalt 200 virtual machines were described as generally available.
  • Guest RDMA was announced as a preview for VM-to-VM communication.
  • Blob Storage Turbo Tier over RDMA targets faster storage access with lower CPU use.
  • The agent infrastructure approach combines isolated execution, identity, storage and secure connectivity.

Azure is bringing compute, storage, networking and security capabilities together to support AI workloads from model training and inference to agent-driven applications. The changes matter to Azure architects, infrastructure teams and developers planning to run AI workloads in production. Microsoft Events demonstrates the infrastructure direction and related capabilities in a Microsoft Ignite session.

What's new

Azure's infrastructure strategy treats AI as more than a GPU provisioning problem. Training, inference and agentic AI place different demands on hardware and operations, so the platform investments span compute, networking, storage and workload management. The aim is to make these components work as a connected foundation rather than as isolated resources.

Training emphasizes GPU capacity, high-bandwidth networking, data pipelines and checkpointing. Inference shifts attention to latency, throughput, model loading and cost per request. Agent workloads add requirements for balancing CPU and GPU resources, connecting securely to tools and data, and maintaining context and task state.

That distinction helps teams think about where infrastructure constraints arise. A model endpoint may need faster startup and better GPU utilization, while a multi-step agent workflow also depends on identity, isolation, networking and persistent information. Azure is presenting its investments as support for the full workload, not just model execution.

Who it is for and what you need

The announcements are aimed at organizations building or operating AI systems on Azure, especially teams moving workloads beyond experiments into production. Infrastructure and platform teams will care about capacity, performance, security and operations; developers will care about how the underlying services support their applications and agents.

The session positions Microsoft Foundry and Azure Kubernetes Service (AKS) as different ways to run applications, with managed simplicity on one side and more direct control on the other. It also describes Azure Container Instances (ACI) as an isolated execution option for agent infrastructure. Which approach fits depends on the workload and the control its operators need.

Specific capabilities have different rollout states. Cobalt 200 virtual machines were described as generally available, while Guest RDMA was announced as a preview. Check the relevant Azure service information and your organization's requirements before planning a deployment around a particular capability.

How it works

The infrastructure story starts by matching the workload to its bottleneck. For inference, repeatedly loading model weights from storage can leave GPUs waiting during endpoint startup or scale-out. A shared cache using NVMe drives is described as a way to let subsequent GPU nodes retrieve already-cached model weights, reducing repeated loading work.

Other announced capabilities address communication and storage access. Guest RDMA lets one virtual machine communicate with another while bypassing the TCP/IP stack, with the goal of reducing latency. Blob Storage Turbo Tier over RDMA is intended to speed access to backend storage; the session describes around 95% lower CPU use per gigabyte.

For agent applications, the session outlines a set of infrastructure building blocks rather than a single service. Isolated execution can use AKS or ACI, Microsoft Entra Agent ID supports authorization and audit, Azure Blob and File Storage can persist agent state, and Agent Grid provides secure connectivity.

  1. Classify the workload as training, inference, agentic AI or a combination, then identify its main constraint.
  2. Choose the hosting approach, such as managed Foundry workloads or AKS when direct infrastructure control is needed.
  3. For inference, evaluate model loading, GPU utilization, storage access and serving latency together.
  4. For agents, plan isolated execution, identity and authorization, state persistence, and controlled connectivity to tools and data.
  5. Confirm each capability's rollout status and suitability before making it part of a production design.

This workload-first approach is useful because improvements in one layer do not automatically resolve issues elsewhere. Faster storage may reduce model-loading delays, for example, but agent workloads still need secure access to their tools and data.

Limitations and alternatives

These capabilities address different parts of an AI system and should not be treated as interchangeable. Guest RDMA targets VM-to-VM communication, while the shared cache and Turbo Tier address storage and model-loading concerns. Agent Grid and identity integration address connectivity and authorization for agents.

Rollout status also matters: the session identifies Guest RDMA as a preview, unlike Cobalt 200 VMs, which it calls generally available. Teams should account for that distinction when evaluating production dependencies. The session presents multiple deployment choices, including Foundry and AKS, rather than prescribing one setup for every application.

Before adopting an option, map it to a measurable workload need: startup time, latency, throughput, cost per request, or controlled agent access. Also consider how the application will preserve context and govern agent actions. The infrastructure decisions should follow those requirements, rather than assuming that adding more GPU capacity alone will solve the problem.

Sources

Frequently asked questions

What AI workloads does Azure's infrastructure target?

The session groups them into training, inference and agentic AI. Each has different infrastructure priorities, from GPU throughput to cost per request and secure agent connectivity.

Is Azure Guest RDMA generally available?

Guest RDMA was described as a preview. The session explains it as VM-to-VM communication that bypasses the TCP/IP stack.

What Azure services can support agent execution?

The session describes AKS or ACI for execution, Microsoft Entra Agent ID for authorization and audit, Azure Blob and File Storage for state, and Agent Grid for secure connectivity.

What is Azure Blob Storage Turbo Tier over RDMA for?

It is intended to provide fast access to backend storage. The session describes around 95% lower CPU use per gigabyte.