AI
LMs can evaluate other LMs better when given more thinking time, study shows
Researchers demonstrate that language models equipped with chain-of-thought reasoning improve at assessing other models' outputs as they generate more intermediate reasoning steps.
As language models produce increasingly fluent and convincing outputs, evaluating their quality has grown into a genuine challenge. A new study from arXiv investigates whether the same principle that helps models solve harder problems—spending more time "thinking"—can also make them better judges of other models' work.
The core finding#
Researchers tested whether reasoning models (LMs that generate explicit step-by-step reasoning chains) could serve as more effective evaluators when given additional test-time compute. The team found that performance improved steadily as these models generated more reasoning tokens, mirroring improvements seen during problem-solving itself.
Two evaluation approaches#
The study examined two complementary evaluation strategies:
- Outcome evaluation: judging the final response as a whole
- Process evaluation: assessing the correctness of each intermediate step separately
By prompting reasoning models to employ both methods, the team leveraged greater test-time compute to enhance assessment accuracy.
Practical implications#
In experiments, reranking multiple model generations using these improved evaluators proved as effective as allocating extra computation to the generation process itself. This finding suggests a new efficiency trade-off: rather than only investing compute in making better responses, practitioners can dedicate similar resources to evaluating existing outputs more thoroughly. The results align with broader trends showing that extended reasoning time consistently benefits language model capabilities across diverse tasks.
