Faster AI Reasoning: New Framework Eliminates Extra Reward Models Without Losing Performance

Large language models are getting better at reasoning through multi-step problems, but the standard approach to teach them—using separate reward models to score each intermediate step—creates significant computational costs. A new framework called Self-Guided Process Reward Optimization (SPRO) addresses this challenge by deriving process rewards directly from the model being trained, dramatically reducing overhead while maintaining accuracy gains.

How Process Reinforcement Learning Works#

Process Reinforcement Learning (PRL) has emerged as a promising approach to enhance LLM reasoning by rewarding not just final answers but the quality of intermediate steps along the way. This step-level feedback, theoretically, should guide models toward more reliable reasoning. However, implementing this scheme typically requires maintaining an additional reward model in parallel with the main policy model—a costly arrangement in terms of memory and computation.

SPRO's Core Innovation#

The research introduces two key advances: (1) a method for extracting process rewards from the policy model itself without external components, and (2) a rigorous mathematical framework for step-wise advantage estimation through Cumulative Process Rewards (CPR) and Masked Step Advantage (MSA). This enables training to evaluate each reasoning step fairly within groups of responses generated from the same prompt.

Performance Gains and Efficiency#

In experiments, SPRO outperformed GRPO (a baseline outcome-only RL method) with 3.4x faster training and a 12.9% improvement in test accuracy. The method also preserved policy exploration throughout training, preventing the model from gaming the reward system with artificially short responses. Crucially, SPRO adds no extra computational overhead compared to simpler outcome-supervised approaches, making it practical for real-world deployment.

Why It Matters#

By eliminating the need for auxiliary reward models while maintaining rigorous theoretical grounding for step-wise feedback, SPRO removes a significant barrier to scaling process-aware training in LLMs. This could accelerate adoption of more sophisticated reasoning training in industrial settings.