New KV Cache Compression Method Cuts Long-Context Inference Costs by 85%

A research team has developed ResidualKV, a compression approach that dramatically reduces the memory and computational overhead of processing long contexts in large language models while maintaining output quality.

The Problem with Long-Context Inference#

Processing lengthy inputs in language models faces a dual challenge. The key-value (KV) cache—a lookup structure that stores attention weights across tokens—grows linearly with context length, consuming enormous amounts of memory. Meanwhile, the attention mechanism itself requires quadratic computation relative to sequence length, making scaling prohibitively expensive. Most existing solutions work around these constraints by either permanently discarding tokens, retaining the full cache, or reconstructing histories on demand—all approaches that compromise either accuracy or efficiency.

How ResidualKV Works#

The method exploits two key observations about how language models process information: tokens at distant positions often encode redundant information, and the differences between token representations follow smooth, predictable patterns. Building on these insights, ResidualKV factorizes the KV cache into two components:

• A sparse set of global reference tokens that can be retrieved dynamically from any point in the sequence
• Compact, quantized residual codes that capture what makes remaining tokens unique without storing full information

When attention needs a token's representation, the system reconstructs it on-the-fly using the reference plus its residual code, avoiding permanent token loss. A further refinement called dynamic-stride scheduling compresses reference growth to approximately logarithmic rates on ultra-long contexts, moving away from linear scaling.

Performance Results#

Testing across multiple model families—Llama, Qwen, LLaVA-OV, and Qwen3-VL—shows substantial efficiency gains. On the LongBench benchmark, ResidualKV achieves near full-cache performance while using only 13-16% of typical KV storage and requiring just 30% of standard attention computation. In multimodal scenarios under matched computational budgets, savings reach 8-10% storage and 10% computation.