Agentic AI
Microsoft Details LLM Routing Strategy for Multi-Model AI Agents on Azure Kubernetes
A new reference architecture shows how to intelligently route AI agent requests across multiple language models on AKS, cutting costs by directing simple tasks to cheaper models.
InfoQ
Microsoft has released a reference architecture for managing AI agent workloads on Azure Kubernetes Service by routing requests across different language models based on task complexity.
Why Intelligent Routing Matters for Agents#
Traditional AI agents execute repetitive cycles—they plan, act, observe, and repeat—often triggering hundreds of LLM calls for a single task. Yet not every call requires a powerful (and expensive) frontier model. Many operations—parsing tool arguments, yes/no decisions, text summarization—work fine with cheaper alternatives. Sending everything to top-tier models wastes money and introduces bottlenecks: a single request with massive token usage can block the entire queue on one GPU while others sit idle nearby.
The Three-Layer Architecture#
The design separates routing concerns into three distinct layers:
-
Semantic routing via RouteLLM: Analyzes prompts using a matrix-factorization model trained on human preferences to predict whether a cheaper model can match a stronger one's output quality.
-
Policy and governance via agentgateway: An OpenAI-compatible proxy that handles authentication, per-agent rate limits, cost tracking, and safety guardrails without needing to understand prompt meaning.
-
GPU-aware placement via the Kubernetes Gateway API Inference Extension: Monitors live GPU state—vLLM's KV-cache occupancy and queue depth—to pick the best replica for each request.
All three converge on a single OpenAI-compatible endpoint. The architecture uses KAITO to provision GPU capacity and run vLLM, while the Endpoint Picker connects to the semantic router directly, bypassing the need for a separate gateway component.
Cost and Quality Trade-Offs#
The critical metric is RouteLLM's escalation threshold—the confidence level that triggers use of the expensive model. In Microsoft's tests, the semantic router achieved 95% of GPT-4's benchmark quality while routing only 26% of calls to the strong model, yielding up to 85% cost savings compared to always using the frontier model.
