Agentic AI
How llm-d Scales Language Model Inference Across Kubernetes Without Breaking Cache Efficiency
Red Hat's distributed inference system extends vLLM across clusters, balancing request stickiness for cached context with load distribution—unlike traditional microservices patterns.
InfoQ
Language model serving on Kubernetes presents a unique challenge: traditional load balancing strategies clash with how modern LLMs actually work. In a recent episode, Edoardo Vacchi from Red Hat discussed how llm-d—a distributed inference framework built on top of vLLM—navigates this tension.
Why Standard Kubernetes Load Balancing Fails for LLMs#
Microservices typically prioritize stateless design: routing can scatter requests across replicas because each instance holds no knowledge of previous interactions. LLM inference inverts this principle. When serving conversational or agentic workloads, routing the same user's follow-up request to a different instance discards valuable cached context—the KV (key-value) and prefix caches that make inference fast and efficient.
Vacchi explained that staying with a single instance maximizes cache reuse and dramatically improves performance, but doing so indefinitely causes uneven load and GPU underutilization. The problem becomes: how do you reap the benefits of inference caching while distributing work fairly across the cluster?
llm-d's Multi-Layered Solution#
llm-d addresses this by introducing several techniques:
- Smart request routing: Prioritizes keeping requests on the same vLLM instance when the cache is warm, only moving them when that instance approaches resource limits
- Batch inference during idle periods: Queues and processes lower-priority requests when the cluster has spare capacity, improving overall throughput
- Semantic classification: Routes prompts to appropriately sized models based on complexity, cutting costs without sacrificing privacy or performance
The vLLM Foundation#
vLLM itself handles high-throughput inference on a single machine through PagedAttention—a GPU memory management technique inspired by systems-level paging. This allows multiple requests to be continuously batched together and served in parallel. llm-d extends this efficiency across multiple vLLM instances in a Kubernetes cluster, solving the scaling problem vLLM alone cannot address.

