New fine-tuning method preserves exploration in reasoning models while maintaining performance

A team of machine learning researchers has developed a novel approach to training large reasoning models that addresses a fundamental tension in post-training: how to imitate expert behavior while preserving the model's ability to explore new solutions.

The Core Problem#

Traditional training pipelines for advanced reasoning models rely on supervised fine-tuning followed by reinforcement learning (SFT-then-RL). During the SFT phase, models learn by copying expert demonstrations, but this process often backfires—the models become overconfident and generate less diverse outputs. When reinforcement learning kicks in later, it must work with this narrowed solution space, limiting potential improvements.

Attempts to fix this with entropy regularization during SFT haven't worked well. Adding entropy penalties simply pushes token distributions toward uniformity, inflating entropy metrics without actually improving the model's capacity for meaningful exploration.

How CurioSFT Works#

The researchers propose CurioSFT, an entropy-preserving method designed to encourage exploration through intrinsic curiosity. The approach consists of two key components:

• Self-Exploratory Distillation: The model learns from a self-generated teacher that uses temperature scaling to encourage exploration beyond its initial behavior. Instead of passively imitating experts, the model distills knowledge from its own temperature-adjusted outputs, promoting creative reasoning paths.

• Entropy-Guided Temperature Selection: This component adaptively tunes distillation strength to balance two competing goals—preserving exploration capability at tokens involved in reasoning while stabilizing factual tokens to prevent knowledge loss.

Demonstrated Results#

Testing on mathematical reasoning benchmarks showed measurable gains. In the supervised fine-tuning stage alone, CurioSFT outperformed standard SFT by 2.5 percentage points on in-distribution tasks and 2.9 points on out-of-distribution benchmarks. More importantly, the exploration capabilities preserved during SFT translated into concrete downstream improvements, yielding an average 5.0-point boost when reinforcement learning was applied afterward.

The findings suggest that designing training procedures around exploration capability—rather than fighting it—can unlock better performance in both imitation and reward-based learning phases.