New Method Tackles LLM Training Collapse by Strategically Mixing Past and Present Reasoning

Large language models refined through reinforcement learning often hit a stubborn problem: as training proceeds, the model's tendency to explore new solutions narrows sharply, and it loses touch with the creative reasoning paths it discovered earlier. This collapse in diversity undermines both exploration and final performance.

The Core Challenge#

On-policy RL training—where the model learns only from outputs it generates itself—gradually exhausts the model's exploratory capacity. Once this diversity vanishes, the model becomes stuck in narrow patterns. Adding historical data to restore variety sounds appealing, but naive approaches that simply blend old trajectories with new ones introduce instability. The policy that generated the old data no longer matches the current model, causing training to degrade.

How Thinking Seeds Works#

The key insight is that the location within a reasoning sequence matters as much as whether to include old data at all. Thinking Seeds uses the model's own earlier checkpoints to seed the beginning of reasoning chains—supplying diverse starting configurations—while letting the model generate the continuation on-policy. This split harnesses historical diversity exactly where it helps most without compromising gradient quality or stability.

The approach works at the token level, using importance weighting to balance off-policy prefixes from the past against on-policy continuations in the present. This granular control keeps training stable while meaningfully expanding the space of solutions the model can discover.

Results and Impact#

Testing across multiple models and math benchmarks shows that Thinking Seeds outperforms standard on-policy training and existing off-policy extensions. Analysis reveals the method maintains stronger effective entropy during training, reduces loss from gradient clipping, and opens up a wider range of explorable solutions—clarifying how position-aware mixing of past and present improves both exploration quality and overall model performance.

This work highlights a broader principle: incorporating historical diversity is not just about whether to use it, but precisely where and how to integrate it into the training process.