OpenRM Uses External Tools to Judge Complex AI Responses Better Than Standard Reward Models

Evaluating whether a language model has produced a correct and useful response is hard, especially when the answer requires knowledge or verification beyond what the model already knows. Reward models have become central to aligning LLMs with human values, but they typically struggle with knowledge-intensive and longer-form outputs where subtle quality distinctions matter. Researchers have introduced OpenRM, a reward model that sidesteps this limitation by calling external tools to gather evidence before making a judgment.

The Core Problem#

Current reward models act as proxies for human evaluation at training and inference time. However, they face blind spots: they cannot reliably discriminate between subtle differences in quality when an answer requires grounding in external facts. This weakness becomes acute for open-ended tasks where correctness hinges on information outside the model's training data.

How OpenRM Works#

Rather than rendering verdicts from internal knowledge alone, OpenRM learns to invoke external tools—such as search engines or fact-checkers—to collect relevant evidence before scoring a response. The model was trained on over 27,000 synthesized pairwise examples using a controllable data synthesis framework. Training employed Group Relative Policy Optimization (GRPO), a technique that jointly supervised both the intermediate tool-calling decisions and the final accuracy assessment. This dual focus encourages the reward model to develop sound evidence-gathering and reasoning strategies.

Empirical Results and Applications#

Testing across three newly-collected datasets and two established benchmarks showed OpenRM substantially outperforming existing reward modeling methods. The researchers then embedded OpenRM into two practical workflows: selecting the best response at inference time and curating training data during the alignment phase. Both applications yielded consistent improvements in downstream LLM alignment tasks.

Why It Matters#

Tool-augmented reward models point toward a pathway for scaling reliable evaluation of complex, knowledge-dependent outputs. By anchoring judgments in external evidence rather than relying solely on the model's internal parameters, OpenRM reduces hallucination and improves the fidelity of human-aligned training signals.