RL Framework Learns Smarter Token Eviction to Cut LLM Inference Memory Costs

Managing the Key-Value cache during large language model inference remains a major bottleneck. As models grow, the memory footprint of cached tokens ballooms—yet existing methods for deciding which tokens to keep rely on crude proxies like recency or historical attention patterns rather than directly predicting whether a token will prove useful downstream.

Researchers have now reframed the eviction problem as a reinforcement learning challenge: instead of following fixed heuristics, train per-head agents to rank tokens by their anticipated value for future generation steps. Their framework, KV Policy (KVP), uses lightweight RL agents that learn specialized eviction strategies from offline generation traces, scoring tokens solely from their key and value vectors—requiring no tweaks to the underlying model.

How the Approach Works#

The method operates by training independent RL agents for each attention head on pre-recorded model outputs. Each agent learns to assign a priority score to cached tokens using a reward signal derived from future utility—essentially learning which tokens will matter most for subsequent decoding steps. Crucially, this learning phase happens offline and demands no architectural changes; the agents integrate into inference without additional computational strain.

Evaluation and Results#

The team benchmarked KVP against strong baselines on demanding long-context tasks: the RULER benchmark reaching up to 128K tokens and the multi-turn dialogue dataset OASST2-4k. Across both settings, the learned eviction policies substantially outperformed heuristic-driven alternatives.

The framework also demonstrated generalization beyond its training regime. In zero-shot trials on downstream tasks like BoolQ, LongBench passage retrieval, and GovReport, KVP maintained strong performance even when handling sequences considerably longer than those seen during training—suggesting the learned utility signals capture robust patterns rather than overfitting to specific scenarios.

Why This Matters#

By shifting from hand-tuned rules to learned predictors of token value, the approach sidesteps the guesswork inherent in heuristic cache management. The results suggest that forecasting future utility—rather than relying on proxies—can be both scalable and generalizable, potentially opening new avenues for adaptive memory management in ever-larger models.