AI
Study reveals flawed human evaluation practices distort AI text-generation rankings
Researchers find that standard human rating protocols fail to capture actual preferences, sometimes reversing true rankings of language models.
Human evaluation is often treated as the definitive measure for assessing natural language generation systems. Researchers analyzing this practice, however, have uncovered a troubling disconnect: the standard protocol used to collect and aggregate human ratings doesn't reliably reflect what people actually prefer.
The fundamental problem#
When evaluating generated text, the typical approach involves collecting ratings from multiple annotators, averaging their scores, and using those averages to rank different systems. A recent arXiv study from May 2022 examines this methodology through the lens of utility theory—a framework from economics—and identifies hidden assumptions baked into the process. Many of these assumptions break down in real-world scenarios, causing the ratings to diverge from annotators' true preferences.
The most problematic culprit appears to be Likert scales (the numbered response options commonly used in surveys). The research demonstrates that these scales can actually flip the direction of preferences in certain cases—meaning systems ranked as better by the numbers might actually be worse according to what evaluators truly prefer.
A proposed remedy and its limits#
The researchers suggest modifications to the standard protocol to make it more theoretically sound. Yet even an improved version faces a hard ceiling: it cannot work well for open-ended, generative tasks like story creation, where outputs vary widely in quality and style.
A new approach for open-ended tasks#
To address this limitation, the team proposes system-level probabilistic assessment (SPA), a fresh human evaluation protocol designed specifically for complex, narrative generation. Testing this method on GPT-3 models of different sizes reveals promising results: SPA successfully recovers the expected ordering (larger models perform better) with statistically significant findings.
The contrast is stark. When evaluated using the traditional protocol on stories, the researchers recovered less than half of the expected preference patterns—for instance, they found no meaningful difference between GPT-3's Curie and DaVinci variants, despite using a well-powered statistical test that should have detected a real gap.
