Study reveals how inference-time sampling with LLM judges balances computational gains against reward misspecification

Large language models increasingly shift computational budget from training time to the inference phase—generating multiple candidate outputs and selecting among them. Yet the underlying principles governing when this strategy succeeds remain murky. A new theoretical and empirical study unpacks the mechanics of this "LLM-as-a-judge" approach and its limits.

The Core Model#

Researchers developed an analytically solvable framework based on Bayesian linear regression paired with a reward-weighted sampler. In their setup, a smaller "judge" model scores candidate outputs using a linear reward function, and the system selects the highest-scoring result via softmax at a chosen temperature. By working in the high-dimensional regime, they derived closed-form expressions for prediction accuracy and error rates.

When More Sampling Helps—and Hurts#

The analysis reveals a critical trade-off. When the judge's reward aligns reasonably well with an underlying ground-truth teacher model, generating additional inference-time samples monotonically improves accuracy. However, if the reward function diverges too far from the true signal—what the team calls reward misspecification—performance hits an optimal sample count beyond which generating more candidates actually degrades results. Crucially, the reward that minimizes error on average generally differs from the teacher itself, so practitioners cannot simply use the teacher as a judge.

Scaling Laws and Temperature Tuning#

For the best-of-k selection scenario using the true teacher as a reward, error decreases as Θ(1/k²), with extreme value theory pinning down the exact prefactor. The authors also show that for any fixed number of samples, an optimal sampling temperature exists that trades off diversity and exploitation.

Task Difficulty Matters#

As task difficulty rises, the computational advantage of inference-time scaling erodes relative to collecting additional training data. This finding delineates practical regimes where sampling many outputs at test time outperforms gathering more labeled examples, and where the opposite holds.

The team validated their theoretical predictions experimentally using large language models with another LLM acting as judge, confirming the theoretical predictions in realistic settings.