LLM Judges Show Consistent Position Bias When Scoring by Rubric

Language models are increasingly deployed as automated evaluators to score text and assess quality, yet a new study from arXiv reveals a hidden flaw in how they operate when following rubrics. Researchers have uncovered that rubric-based evaluation inherently triggers position bias—the tendency for LLMs to disproportionately favor score options appearing at specific locations within a rubric's scoring scale.

How Position Bias Manifests#

The research demonstrates that rubric-based scoring functions like a multiple-choice task, exposing LLMs to systematic positional preferences. Across controlled experiments spanning multiple models and datasets, the position bias emerged as a consistent pattern. Crucially, the direction of bias proved model-specific: some judges gravitationally pull toward the highest-ranked option at the top of the list, while others lean toward the final position at the bottom.

Beyond simple positional preferences, the researchers identified a secondary, independent axis of bias. When a single prompt asks an LLM to evaluate multiple criteria simultaneously, the sequence in which those criteria are presented shifts the final scores, compounding the evaluation distortion.

Testing Mitigation Strategies#

To counter these biases, the team explored randomizing the order of rubric options—shuffling their arrangement across multiple evaluation runs. While this approach did successfully reduce position bias, the practical gains were mixed. Models exhibiting strong initial bias saw meaningful improvements in alignment with human annotations, but benefits remained modest across the broader model landscape. Reassuringly, the analysis showed that only a handful of random permutations are needed for most models to meaningfully diminish the error introduced by positional preferences.

Implications for LLM-as-Judge Systems#

These findings reframe rubric-based LLM evaluation as a multiple-choice problem with measurable, predictable distortions. For practitioners deploying language models as judges—especially in domains where human verification is prohibitively difficult—the work underscores the importance of understanding model-specific bias patterns and implementing order randomization as a lightweight safeguard.