Study Reveals Hidden Memorization Shortcut in LLMs Trained with Flawed Rewards

A new arXiv study investigates a puzzling phenomenon in large language models: when trained with Reinforcement Learning using Verifiable Rewards (RLVR), models like Qwen 2.5 show performance gains even when the reward signals are incorrect or misleading.

Researchers analyzed this counterintuitive result and uncovered what they call the "Perplexity Paradox." The mechanism works like this: spurious rewards cause the model's perplexity on answer tokens to drop sharply while coherence on the prompt side degrades, suggesting the model bypasses genuine reasoning and instead retrieves memorized solutions.

Finding the Circuit#

Using interpretability tools including Path Patching, Logit Lens, JSD analysis, and Neural Differential Equations, the team identified a hidden "Anchor-Adapter circuit" that enables this memorization shortcut. They found:

• Functional Anchor: Located in middle layers (L18-20), this component triggers retrieval of stored answers
• Structural Adapters: Found in later layers (L21+), these reshape representations to accommodate the shortcut signal

The researchers also demonstrated that by scaling specific MLP keys within this circuit, they could artificially amplify or suppress the contamination-driven performance—essentially steering the model's behavior in either direction.

Implications#

The findings provide a mechanistic roadmap for identifying and mitigating data contamination in RLVR-tuned models, highlighting a critical vulnerability in reward-based fine-tuning approaches. Code for the research is available alongside the paper.