New Reward Modeling Approach Boosts LLM Agent Performance Beyond Fine-Tuning

Researchers have identified a more effective path to improving how well language model agents perform on unfamiliar tasks: rather than directly refining the policy model itself, guiding it with a separately trained reward model yields superior results across diverse benchmarks.

The Generalization Challenge#

Large language model-based agents have become increasingly capable on tasks they were trained on, yet their ability to transfer those skills to novel assignments remains limited. Past efforts to address this gap have typically focused on fine-tuning the policy model—the core decision-making component—using a broader range of training examples. This team's investigation reveals a different approach is more reliable: training a dedicated reward model to steer the policy model's choices.

Introducing AgentRM#

The researchers developed AgentRM, a generalizable reward model designed to guide policy generation during testing. They explored three distinct methods for building such a model: direct reward scoring, implicit reward inference, and leveraging LLMs as judges. Once constructed, AgentRM directs answer generation using two techniques—Best-of-N sampling to pick the strongest candidate from multiple options, and step-level beam search to optimize the reasoning path.

Performance Gains#

Testing across nine agent tasks spanning four categories, AgentRM lifted baseline performance by an average of 8.8 points, outpacing leading general-purpose agents by 4.0 points. The approach also demonstrated "weak-to-strong" generalization, where applying it to a larger model (LLaMA-3-70B policy) generated even larger improvements of 12.6 points. Additionally, it proved effective as a complement to already fine-tuned models, exceeding top specialized agents by 11.4 points on held-out test tasks.

Why It Matters#

The findings suggest that separating the reward signal from direct policy optimization offers a more robust path to agent generalization and enables efficient test-time scaling without requiring repeated retraining.