New Diversity-Based Exploration Method Helps Language Models Discover Novel Behaviors

Reinforcement learning (RL) can unlock new capabilities in language models, but a key question remains: are current techniques truly discovering fresh behaviors, or merely refining what's already embedded in the base model? A new paper tackles this challenge by investigating how deliberate, guided exploration can help models venture beyond their initial abilities.

The Core Insight: Representation-Based Diversity#

The research centers on a simple but effective idea: reward language models explicitly for discovering diverse and novel approaches. The key innovation is a representation-based bonus derived from the model's internal hidden states — a mathematically grounded way to measure and incentivize behavioral diversity. By tapping into the knowledge already present in pre-trained models, this method guides the search toward genuinely new behaviors rather than random exploration.

Dual Benefits: Inference and Post-Training#

The approach shows gains in two complementary settings:

• At test time: Exploration with representation-based diversity makes models more efficient at reasoning tasks. For instance, on Qwen-2.5-14b-Instruct, the method achieved over 50% improvement in verifier efficiency across nearly all tested reasoning benchmarks.

• During post-training: When integrated into an RL pipeline, the exploration strategy surpasses both the baseline model and standard RL approaches. On AIME 2024, a post-trained Qwen-2.5-7b-Instruct reached the performance level of a competitor's model at 3x higher sample efficiency (pass@80 matching pass@256).

Why This Matters#

The findings suggest that smart, principled exploration—rather than brute-force sampling—is the path forward for pushing language models beyond their inherent constraints. By grounding diversity in the model's learned representations, the technique achieves a practical balance between leveraging existing knowledge and discovering genuinely new reasoning strategies.