How to Measure LLM Judge Reliability: A Framework for Consistent Evaluation Reporting

When researchers use large language models to score or evaluate text in place of human reviewers, they typically measure success by comparing the LLM's verdicts to human labels. But a new paper on arXiv shows that the same underlying judgments can yield wildly different agreement scores depending on decisions made before any statistical test is even run.

The Hidden Variability in Measuring Agreement#

The core problem is that agreement metrics are not as standardized in machine learning evaluation as they are in established fields like psychometrics or corpus linguistics. How should researchers handle edge cases like missing judgments from the LLM? Should they count all items or exclude uncertain cases? What judgment scale should they use—binary yes/no, or a richer scale? These seemingly minor protocol choices turn out to have major consequences.

The researchers analyzed three published LLM-judge studies and found that the same verdicts, analyzed under different (but defensible) protocols, could produce accuracy figures ranging from 55% to 90%—a 35-point swing. In one benchmark example, changing the measurement protocol even made statistical significance flip: Cohen's kappa moved from positive to below zero without any actual judge verdict changing.

What the Statistics Actually Show#

The paper reveals surprising redundancy in common agreement metrics. For straightforward binary judgments, Pearson's correlation, Spearman's rank correlation, Kendall's tau, the phi coefficient, and Matthews correlation coefficient all compute exactly the same number, even though researchers often report them as distinct measures. Cohen's kappa differs mainly by a correction factor tied to how often each rater chose the positive label—and they share the same statistical properties when judges and humans make positive calls equally often.

This means many published comparisons report the same statistic multiple times under different names, creating a misleading impression of independent confirmation.

A Recipe for Reproducible Reporting#

To address the problem, the authors propose a measurement protocol checklist that researchers should specify when reporting agreement numbers. This checklist includes decisions about:

• Judgment scale type (binary, ordinal, etc.)