RequestRouter accelerates single-GPU AI inference by 2x with dynamic request-level optimization

A new system called RequestRouter offers a practical way to speed up large language model inference on a single GPU by dynamically choosing how to process each request based on its characteristics—without requiring model retraining or hardware upgrades.

How it works#

Rather than running all requests through the same inference setup, RequestRouter evaluates cheap, lightweight features of each incoming request and assigns it to one of several pre-configured inference modes. These modes include standard FP16 (16-bit floating point), quantized inference, speculative decoding, prefix caching, continuous batching, and hybrid combinations such as GPTQ with prefix caching or INT8 with continuous batching.

The router operates at the request boundary—meaning it makes its decision once per request—keeping computational overhead minimal. The system was evaluated using an 8B instruction-tuned language model served through vLLM on NVIDIA A100 GPUs.

Performance gains#

Across extensive testing involving 30,000 measured inference executions, RequestRouter achieved a 2.10x mean latency speedup compared to standard FP16 baseline performance on deployment-style workloads, while reducing energy consumption to 0.48x of the baseline. A tightly controlled statistical evaluation confirmed these improvements persist: the system maintained a 1.93x latency speedup (95% CI: 1.88–1.98x) with a 0.523 energy ratio (95% CI: 0.506–0.540).

Accuracy remained largely intact across the board, with the routed policy preserving 99.6% of FP16 macro accuracy in expanded benchmark testing. The routing overhead itself is negligible—a CPU microbenchmark of 100,000 calls measured only 0.00475 ms mean latency per routing decision, with a p99 of 0.00532 ms.

Why it matters#

The approach addresses a practical constraint for many deployments: not all organizations have access to large GPU clusters or can afford expensive model retraining. By intelligently pairing requests with the most efficient inference path on existing single-GPU setups, RequestRouter delivers substantial performance and energy benefits with minimal implementation complexity.