New Batching Strategy Boosts LLM Inference Throughput by Grouping Similar-Length Requests

A team of researchers has introduced a scheduling technique designed to squeeze more efficiency from large language model inference systems, tackling a fundamental bottleneck in how servers batch and process multiple requests simultaneously.

The Problem with Mixed-Length Requests#

When multiple LLM queries run in parallel on the same hardware (like GPUs), the entire batch must wait for its slowest member to finish before the system can move on. This creates a significant drag on overall throughput: if one request happens to need a lengthy output while others are quick, the server's resources sit idle waiting for completion. This mismatch between fast and slow requests leads to wasted computational capacity that could otherwise be serving new queries.

Multi-Bin Batching Approach#

The researchers formalize this challenge as a queuing problem and propose Multi-Bin Batching as a solution. The method works by predicting how long each incoming request will take to process, then sorting requests into predetermined "bins" based on their estimated execution time. Requests within the same bin—having comparable lengths—are batched together, reducing idle time and maximizing hardware utilization.

This straightforward idea comes with theoretical backing: the team demonstrates through queueing theory that their approach is throughput-optimal under a static-batching framework, where batches are locked in and processed sequentially.

Performance Gains and Limitations#

Empirical testing across real-world LLM inference workloads shows that multi-bin batching substantially outperforms conventional static batching, where requests are grouped without regard to execution time. However, the method still leaves a measurable gap compared to continuous batching—an approach where new requests can join ongoing batches dynamically. The researchers quantify this remaining performance difference under both ideal (oracle) and realistic (estimated) length prediction scenarios.

The work bridges the performance spectrum between simpler static batching and more complex continuous techniques, offering a practical middle ground for inference system optimization.