New scheduling approach cuts LLM latency spikes by up to 77% without moving cached data

Researchers have developed Astrolabe, a new method for distributing incoming requests across multiple language model servers more fairly and with lower latency, sidestepping the network overhead that comes with moving cached data between machines.

The Load Balancing Problem#

Large language models running on distributed servers face a fundamental challenge: requests arrive unpredictably and consume different amounts of compute depending on how long their responses need to be. When one server gets overloaded while others sit idle, users experience delays. Existing solutions often shuffle the model's cached values (called KV-cache) from busy servers to idle ones, but this data transfer can itself become a bottleneck under heavy traffic.

How Astrolabe Works#

Astrolabe takes a different approach: instead of rebalancing after requests arrive, it predicts what each server will need before assigning new work. The system estimates how long a response will be, simulates how that would affect each server's latency, and then uses a two-choice random selection method to spread the load. This one-shot dispatch means requests are routed immediately without the delay and network strain of shuffling caches afterward.

Performance Gains#

On a standard benchmark using Llama-2-7B models and real user queries, Astrolabe matched the throughput of the best competing approach while cutting mean time-to-first-token (the time until a user sees the first word) by 8–36 percent and 99th-percentile latency by 16–77 percent. The system also reduced overall response time by up to 5.6 percent and dramatically lowered the number of requests that get preempted once the server reaches capacity—dropping from typical baselines to roughly one-sixth as many interruptions.

In more challenging setups—different model sizes, burst traffic patterns, or tighter batching constraints—Astrolabe improved throughput by 6–7 percent and end-to-end latency by 6–9 percent, while using far less CPU per prediction engine. When cache migration was enabled on high-end GPUs, Astrolabe outpaced a competing system called Llumnix by 2.6 times in raw throughput.

Why It Matters#

Prediction-guided scheduling offers a practical path forward for LLM inference infrastructure. By avoiding expensive cache shuffles and instead making smarter routing decisions upfront, operators can squeeze better performance out of existing hardware without requiring architectural changes or new network capabilities.