New Training Method Guides LLM Reasoning Through Step-by-Step Rewards

Reinforcement learning is increasingly used to strengthen the reasoning capabilities of large language models, but training on only final correct or incorrect answers misses an important opportunity: understanding which intermediate steps led to success. A new arXiv preprint introduces a method to extract detailed learning signals directly from the model itself.

The Problem with Outcome-Only Rewards#

Current approaches typically rely on sparse, binary rewards that only judge whether a final answer is correct. This creates a blind spot: when a model produces the right result, there's no signal indicating which of its intermediate reasoning steps were sound or which were lucky guesses. For multimodal tasks involving images, this approach can inadvertently reward answers driven by text patterns rather than actual visual understanding.

Introducing Stepwise Marginal Information Gain#

Researchers from the arXiv submission propose Stepwise Marginal Information Gain (MIG), a process reward calculated entirely from the model's own outputs during training. The approach works by measuring how each reasoning step changes the likelihood of arriving at the correct answer.

A key innovation is a monotonic historical watermark that prevents the model from receiving duplicate credit: it rewards only new likelihood peaks, avoiding repeated praise for retracing ground already covered during the learning process.

How It Integrates with Training#

The method combines MIG with outcome rewards, format rewards, and a gated self-distillation mechanism that keeps only trajectories that are both structurally sound and correct. For vision-language models, an additional gate compares predictions with and without images, down-weighting rewards when the answer would be predictable from text alone.

Experimental Results#

Testing across eight task-specific benchmarks shows consistent improvements over outcome-only baseline training. In broader transfer scenarios, the approach gains up to 4.8 points in average accuracy over binary-reward methods, and reaches 12.6 points on MathVerse benchmarks. At the 7B model scale, the method outperforms external process reward baselines by 12.9 points on vision-language transfer tasks without needing extra reranking steps during inference.