Reinforcement Learning Tunes Reasoning, Supervised Fine-Tuning Rebuilds It—New Study Shows the Difference

A fresh analysis of how thinking language models acquire reasoning capabilities reveals a striking divide: base models already possess the raw mechanisms for logical reasoning, but training methods differ dramatically in what they teach—whether they're optimizing existing components or installing entirely new ones.

How Reasoning Gains Happen#

Researchers developed an interpretable lens for understanding reasoning by building small neural networks called Sparse Autoencoders on the internal activations that models generate while performing reasoning tasks. This unsupervised approach surfaced distinct categories of reasoning behavior in models without explicit human guidance. They then applied a technique called "constructive model diffing" to trace exactly which reasoning mechanisms activate during problem-solving and which training-time rules govern their use.

Two Paths to Better Reasoning#

The study examined nine different model pairs—some trained with reinforcement learning (RL), others with supervised fine-tuning (SFT), and a hybrid combination. The results were consistent across independent analyses:

• RL-trained models recovered about 76% of the performance gap between base and advanced versions by learning timing rules (heuristics) for mechanisms already present in the base model
• SFT-distilled models recovered only 11% this way, indicating they were fundamentally installing new reasoning structures rather than optimizing existing ones

This distinction has immediate relevance: RL primarily teaches when to apply pre-existing reasoning capabilities, while SFT teaches what new reasoning structures to deploy.

Why This Matters#

The findings suggest a new framework for understanding what different training paradigms accomplish. If building stronger reasoning models requires choosing between teaching orchestration versus installing novel mechanisms, this opens pathways for more efficient model development—potentially avoiding redundant computation or focusing training effort where it's most needed. The research provides a more granular view of the black box inside language models, revealing that capability gains stem from qualitatively different learning processes depending on the training regime.