LLM Code Judges Show Systematic Bias Against Cosmetic Code Variations

Researchers have identified a significant reliability problem with using large language models as automated code evaluators. According to a comprehensive study on arXiv, LLM judges systematically struggle to fairly assess code that is semantically identical but stylistically different—a critical gap for AI-driven code evaluation at scale.

The Problem with LLM Code Judges#

As organizations increasingly rely on LLMs to assess generated code without reference implementations, questions emerge about whether these models can apply consistent standards. In practice, correct code often varies in superficial ways: different variable names, added comments, alternative formatting, or restructured logic—changes that should not affect whether the code actually works. The study shows that all tested LLM judges exhibit both positive and negative biases, sometimes unfairly inflating scores and other times unfairly deflating them.

Six Types of Bias Identified#

The researchers defined and systematically examined six categories of potential bias in how LLMs evaluate code. Across five programming languages and multiple LLM architectures, each model fell prey to these biases, demonstrating that the problem is widespread rather than limited to particular models or languages.

Prompting Alone Doesn't Fix It#

One notable finding challenges a common mitigation strategy: even when LLM judges were prompted to generate test cases before scoring code, they remained vulnerable to these superficial biases. This suggests that simply asking models to "think harder" or follow a better procedure does not address the underlying inconsistency.

Why It Matters#

The implications are significant for anyone deploying LLM-based code evaluation systems. Biased judgments could lead to unfair comparisons between different approaches to the same problem, skewed automated feedback for developers, or compromised benchmarking for code generation models. As LLMs become more central to software development workflows, the need for robust, bias-resistant evaluation methods becomes increasingly urgent.