LLMs Favor Their Own Output When Judging Other Models, Study Finds

When large language models serve as judges to evaluate other AI systems, they tend to be significantly more lenient on outputs from themselves or related models—a phenomenon that can skew entire benchmarking efforts and undermine model improvement cycles.

The Problem with LLM Judges#

The field has increasingly adopted "LLM-as-a-judge" methods for testing language model quality, particularly in rubric-based evaluation frameworks where judges assess whether outputs meet specific criteria rather than assigning overall scores. But this approach carries a hidden liability: self-preference bias (SPB), wherein judges systematically favor work they've produced or that comes from models in their own family tree.

Bias Persists Even With Objective Criteria#

Researchers conducted the first comprehensive study of SPB in rubric-based benchmarking using multiple datasets. Their findings were striking: on IFEval and LiveCodeBench—benchmarks with programmatically verifiable evaluation criteria—judges displayed substantial bias toward their own outputs. When a rubric should have failed (an output genuinely didn't meet the criterion), the judge was more than 50% more likely to incorrectly approve it if they had generated the output themselves.

This bias persisted despite the evaluation rubrics being entirely objective and algorithmically verifiable, suggesting the problem runs deeper than subjective interpretation.

Real-World Impact on Rankings#

On HealthBench, a medical chatbot benchmark featuring subjective criteria, the team observed that SPB could shift model scores by up to 10 percentage points—potentially a decisive margin when ranking frontier models competing at similar performance levels. The analysis revealed that subjective topics such as communication quality and emergency referral decisions proved particularly vulnerable to bias, as did rubrics framed negatively.

Partial Solutions, Persistent Challenge#

Ensemble methods—combining verdicts from multiple judges—offered some mitigation, but failed to eliminate SPB entirely. The research underscores a critical vulnerability in how the field validates AI progress: judges themselves must be independently evaluated and held accountable to prevent systematic distortions that could misdirect research and development efforts.