New algorithm tackles memory bottleneck in large language models by smartly compressing cached data

When large language models generate text, they store previously calculated key-value pairs to speed up computation. This KV cache grows with sequence length and becomes a major memory constraint for deploying models at scale. Compressing this cache through quantization can help, but existing approaches treat all attention heads equally—missing the opportunity to allocate higher precision where it matters most.

The mixed-precision challenge#

A natural refinement would assign more compression bits to important heads and fewer to less critical ones. However, researchers working on RateQuant discovered a subtle trap: different quantization methods follow different compression-to-quality curves, with decay rates ranging from 3.6 to 5.3. If you apply one quantizer's assumptions to optimize bit allocation for another, the allocation order flips and performance actually falls below uniform compression. The team calls this problem distortion model mismatch.

How RateQuant works#

RateQuant resolves this by measuring each quantizer's actual distortion curve from a small calibration dataset, then applying reverse waterfilling—a technique from rate-distortion theory—to solve the bit-allocation problem in closed form. This approach accounts for the specific characteristics of each quantization method and correctly identifies which heads deserve more precision.

Real-world impact#

Testing on Qwen3-8B at 2.5 average bits per value, the calibrated approach slashed perplexity from 49.3 down to 14.9 (a 70% reduction) compared to the baseline KIVI quantizer. It also improved QuaRot by 6.6 perplexity points. The entire calibration runs in 1.6 seconds on a single GPU and adds no overhead during inference—making it practical for production systems.