Targeted Expert Guidance Improves LLM Reasoning Without Sacrificing Diversity

Researchers have proposed a framework to address a persistent challenge in training language models to reason better: helping them explore the space of possible reasoning paths without simply copying expert examples.

The Problem with Current Approaches#

When training large language models to improve their reasoning ability using reinforcement learning with verifiable rewards (RLVR), a fundamental tension arises. The model needs strong enough capabilities to conduct thorough exploration—meaning it must find both effective solutions and diverse ways of reaching them. However, existing methods that guide models by imitating full expert reasoning sequences achieve effectiveness at the cost of diversity: the model learns to mimic surface-level patterns rather than grasping the underlying strategic principles.

A Smarter Intervention Strategy#

The researchers behind MENTOR (Mixed-policy Expert Navigation for Token-level Optimization of Reasoning) introduce a different premise: expert guidance works best when applied sparingly, at moments that matter most. Rather than having the model shadow an expert through an entire reasoning chain, the framework directs intervention only at critical decision points—the junctures where the model's choices have the greatest impact on whether it can reach a correct solution.

This targeted approach allows the model to absorb the logic behind expert strategy rather than defaulting to imitation.

Results and Implications#

Experiments demonstrate that MENTOR enables models to capture the essence of what makes expert reasoning effective, allowing them to conduct higher-quality exploration across diverse solution paths. By focusing guidance where it counts most, the framework sidesteps the effectiveness-diversity trade-off that has limited prior work. The authors have made their code publicly available, supporting reproducibility and broader adoption of the method.

The work suggests that in reinforcement learning for language model reasoning, less can sometimes be more: strategic, sparse intervention outperforms comprehensive supervision.