Scaling AI Agents on AKS: A Deep Dive into Microsoft’s Three-Layer LLM Routing Architecture

The shift from simple chatbots to autonomous AI agents marks a significant transition in how we deploy Large Language Models (LLMs). While a standard RAG (Retrieval-Augmented Generation) application might handle a single request-response pair per user interaction, an agentic system operates in a “Plan-Act-Observe” loop. A single user query can trigger dozens of internal LLM calls as the agent reasons through a problem, executes tools, and validates its own output.

This iterative nature introduces what we call “agentic overhead.” If you are running these agents at scale on Kubernetes, the traditional ways of managing traffic simply fall apart. Standard round-robin load balancing, which works perfectly for microservices, is catastrophic for LLMs. Furthermore, using a frontier model like GPT-4o for every intermediate step—such as formatting a JSON schema or summarizing a tool output—is an economic failure.

To solve this, Microsoft has introduced a reference architecture for Azure Kubernetes Service (AKS) that treats LLM routing as a three-layered problem. By decoupling semantic intent from hardware state, this architecture allows platform engineers to maximize GPU utilization while minimizing costs.

The Agentic Overhead: Why Standard Architectures Fail

When we build agents, we are essentially building loops. Consider a simple research agent:

1. Plan: The agent breaks down the query into steps (LLM call 1).
2. Act: The agent searches for data (Tool call).
3. Observe: The agent reads the results and decides if it needs more info (LLM call 2).
4. Iterate: Steps 2 and 3 repeat potentially 5–10 times.
5. Finalize: The agent synthesizes the final answer (LLM call 3).

In this scenario, one user request has generated 12 LLM interactions. If your architecture treats every one of these calls as a high-priority request to a frontier model, your API costs will skyrocket. More importantly, you face the latency problem.

In standard Kubernetes networking, an Ingress controller or Service load balancer uses Layer 7 (L7) round-robin or least-connections logic. However, LLM inference is unique because the “cost” of a request is not uniform. A “prefill” operation (processing the input tokens) is computationally different from “decoding” (generating new tokens). Standard load balancers are blind to the state of the KV-cache (Key-Value cache) on the GPU. If a load balancer sends a massive 32k-token prompt to a GPU node that is already struggling with high KV-cache occupancy, it causes head-of-line blocking. Small, fast requests get stuck behind massive computations, destroying the user experience.

The Three-Layer Solution: A Reference Architecture

To address these challenges, the Microsoft reference architecture divides the routing responsibility into three distinct layers. This separation ensures that the system understands the meaning of the request, the policy of the enterprise, and the physical state of the hardware.

This tiered approach ensures that a request is vetted for complexity before it ever reaches a GPU, and once it does, it is sent to the specific pod best equipped to handle it at that exact millisecond.

Layer 1: Semantic Routing with RouteLLM

The first gate a request hits is the Semantic Layer. The core idea here is that not all LLM calls are created equal. If an agent needs to “Check if the following text is offensive (Yes/No),” it does not need a trillion-parameter model. A 7B or 14B parameter model running locally on AKS via vLLM is more than sufficient.

Matrix Factorization for Model Prediction

RouteLLM uses a technique called Matrix Factorization to predict how well a “weak” model (like Llama-3-8B) will perform on a specific prompt compared to a “strong” model (like GPT-4o). By analyzing historical performance data and prompt embeddings, RouteLLM can calculate a score. If the predicted performance of the weak model is within a certain threshold (e.g., 95% of the strong model’s capability), the request is routed to the cheaper, self-hosted model.

Balancing the Pareto Frontier

This allows architects to define a cost-performance trade-off. You can tune the router to be “aggressive” (saving more money by using local models) or “conservative” (defaulting to Azure OpenAI for anything remotely complex). This is critical for agentic loops where 80% of the traffic consists of intermediate reasoning steps that don’t require world-class knowledge.

Layer 2: Governance and Policy via agentgateway

Once the destination model type is decided, the request passes through the agentgateway. This layer acts as a centralized proxy that provides an OpenAI-compatible API surface. This is vital for developer experience; your AI engineers don’t need to know the complexities of the underlying Kubernetes cluster—they just point their SDKs at the gateway.

The Role of ‘ext-proc’

The agentgateway is built to integrate deeply with the Kubernetes networking stack. It uses the ext-proc (External Processing) protocol to talk to the infrastructure layer. When a request comes in, the gateway handles:

• Authentication: Ensuring the agent has the rights to call the model.
• Rate Limiting: Preventing a runaway agent loop from consuming the entire monthly budget in ten minutes.
• Cost Tracking: Tagging requests with metadata to attribute costs to specific teams or projects.
• Guardrails: Applying PII redaction or content safety filters before the prompt reaches the inference engine.

By handling these at the gateway level, you ensure consistent policy enforcement across both self-hosted models on AKS and external models like Azure OpenAI.

Layer 3: GPU-Aware Routing with Gateway API Inference Extension

This is where the architecture solves the “head-of-line blocking” problem. Standard Kubernetes load balancing stops at the service level. The Kubernetes Gateway API Inference Extension goes deeper by looking at the real-time metrics of the inference engine (e.g., vLLM or Hugging Face TGI).

KV-Cache and the Endpoint Picker

The most critical metric in LLM inference is KV-cache occupancy. The KV-cache stores the context of current generations. If a pod’s cache is 90% full, any new request will likely trigger “prefill starvation,” where the pod spends all its time swapping data in and out of memory rather than generating tokens.

The Inference Extension includes an Endpoint Picker. Instead of round-robin, the picker queries the pods for:

1. Available KV-cache slots.
2. Current queue length.
3. Current running vs. waiting requests.

Defining the Infrastructure with CRDs

The extension introduces Custom Resource Definitions (CRDs) that allow you to manage LLM capacity as a first-class citizen in Kubernetes:

apiVersion: inference.networking.x-k8s.io/v1alpha1
kind: InferencePool
metadata:
  name: llama-3-70b-pool
spec:
  selector:
    matchLabels:
      app: vllm-llama-3
  strategy:
    type: LeastLatency
---
apiVersion: inference.networking.x-k8s.io/v1alpha1
kind: InferenceObjective
metadata:
  name: optimize-throughput
spec:
  poolRef:
    name: llama-3-70b-pool
  maxLatency: 500ms
  throughputTarget: 1000tps

The InferencePool groups your GPU pods, while the InferenceObjective tells the controller what to prioritize. This level of granularity is what allows AKS to handle the bursty, unpredictable traffic patterns of agentic workloads.

Operationalizing the Stack on AKS

Building this architecture manually would be a nightmare of configuration. Microsoft streamlines this by using KAITO (Kubernetes AI Toolchain Operator). KAITO automates the deployment of large language models on AKS by choosing the right VM size (e.g., NDv4 series with A100/H100 GPUs) and configuring the inference engine.

Telemetry and Monitoring

To make the Endpoint Picker work, you need real-time data. This is achieved by:

1. vLLM Metrics: The inference pods export metrics like vllm:num_requests_running.
2. Azure Managed Prometheus: Scrapes these metrics every few seconds.
3. Gateway API Extension: Subscribes to these metrics to make routing decisions.

Autoscaling

For agentic workloads, you should use the Horizontal Pod Autoscaler (HPA) based on custom metrics rather than CPU/Memory. Scaling on avg_tokens_per_second or kv_cache_usage ensures that your cluster expands before users start seeing “Request Timed Out” errors.

The Economics of Agentic AI: Prompt Caching and Deflation

Implementing this three-layer routing isn’t just about technical elegance; it’s about survival in an era of efficient AI development.

One of the massive benefits of consistent, intelligent routing is prompt caching. When an agent repeats a loop, much of the prompt (the system instructions and the history) remains the same. Modern inference engines can cache these prefixes. By using the Gateway API to route similar requests to the same set of pods, you increase the cache hit rate, which can reduce latency by up to 80% and significantly lower the compute cost per token.

This architecture contributes to the broader AI deflationary spiral in IT outsourcing. As the cost of running complex agents drops, tasks that previously required human-in-the-loop oversight can be fully automated at a fraction of the previous cost. By moving from expensive “Frontier-only” models to a hybrid AKS architecture, enterprises can achieve a much lower Total Cost of Ownership (TCO) for their AI platforms.

“The goal of AI infrastructure is to make the intelligence invisible. When we route a request to a 7B model because it’s the right tool for the job, we aren’t just saving money—we’re building a more resilient system.”

Future Outlook: The Stabilization of AI-Native Infrastructure

We are moving toward a world where Kubernetes is no longer just a container orchestrator, but an “AI orchestrator.” The standardization of the Kubernetes Gateway API Inference Extension is a major step in this direction.

In the near future, we expect to see:

• Predictive Routing: Moving from reactive metrics to proactive scheduling. The router will use ML models to predict how many tokens a request will generate before it starts, allowing for even tighter bin-packing of GPU resources.
• Hardware-Agnostic Scheduling: The ability to seamlessly route between NVIDIA, AMD, and custom silicon (like Microsoft’s Maia chips) based on real-time availability and cost.
• Native Agent Support: Gateways that understand the state of an agent’s “thought process,” keeping the entire loop’s context local to a specific node pool to minimize cross-node data transfer.

For platform engineers, the message is clear: the days of simple load balancing are over. To scale the next generation of agentic AI, you must build a stack that is as intelligent as the models it hosts. By adopting this three-layer architecture on AKS, you provide the governance, efficiency, and performance required to turn experimental agents into production-grade enterprise assets.