New Framework Combines Inverse Reinforcement Learning and Search to Boost LLM Reasoning Performance

Test-time scaling—allowing language models to spend extra computational effort during inference—has emerged as a promising path to improving performance on hard problems. However, existing approaches rely on methods that have significant drawbacks: outcome-based reinforcement learning suffers from instability and inefficient sample use, while static reward models designed to evaluate reasoning steps require expensive manual labeling and can be misled when faced with new types of problems.

A Unified Approach#

Researchers have introduced AIRL-S, a framework designed to sidestep these limitations by combining Adversarial Inverse Reinforcement Learning with Group Relative Policy Optimization. Rather than requiring hand-labeled step-by-step feedback, the system infers a dense reward signal directly from reference trajectories—examples of good reasoning paths. This learned reward model then serves double duty: it guides the model's training and also evaluates candidate solutions during search.

How It Works#

The core innovation lies in learning rewards end-to-end from trajectory data. By inferring what makes each intermediate step valuable, AIRL-S eliminates the need for costly step-level annotation. The same learned model acts as both training objective and verification mechanism during search-based inference, creating a more cohesive pipeline.

Results and Scope#

Evaluation across eight benchmarks spanning mathematics, science, and code generation showed the approach improves average performance by 9% relative to the base model—matching performance of GPT-4o on these tasks. The researchers also examined how the two optimization objectives complement each other and demonstrated that the learned reward model generalizes across different underlying generators and search strategies, suggesting the method's applicability to real-world deployment scenarios.

Implications#

The work establishes a more cost-effective methodology for scaling test-time computation on complex reasoning challenges, potentially reducing the annotation burden that has constrained earlier approaches.