New Scheduling Method Slashes Language Model Serving Latency by Over 15x

Large language models with reasoning capabilities generate unpredictable amounts of output, creating scheduling headaches for services trying to deliver fast responses. When a queue processes requests in order, long-running tasks can trap shorter ones behind them—a problem known as head-of-line blocking.

Researchers have now proposed a solution: PARS, a scheduler that ranks queued tasks by their likely completion time before serving them. By approximating shortest-job-first strategies without measuring execution time, the system cuts latency dramatically across chat, math, and code-generation workloads.

How PARS Works#

The core insight is simple: rather than waiting to see how long a task actually runs, PARS estimates response length directly from the input prompt. It uses pairwise ranking with margin loss—a machine learning technique that learns to order tasks correctly without needing exact time predictions. The system integrates with vLLM, a popular open-source LLM serving platform, so researchers can adopt it immediately.

Real-World Impact#

Testing across multiple language models and inference scenarios shows that PARS reduces latency by up to 15.7x compared to vLLM's default first-come, first-serve approach. The gains hold even when the scheduler is trained on one model and applied to others, suggesting the ranking strategy generalizes across different LLM architectures.

Why It Matters#

As reasoning-capable models become more common—generating variable-length outputs ranging from brief answers to lengthy problem-solving chains—scheduling efficiency becomes a bottleneck for production systems. By turning scheduling into a machine learning problem solvable at admission time, PARS addresses a real pain point without adding per-request overhead.