AI
Researchers identify when AI code judges are actually trustworthy—and when they're just confident
New research reveals that multi-agent code evaluation systems often mask uncertainty with false confidence, but a label-free diagnostic can reveal which verdicts are actually grounded in evidence.
When one language model evaluates whether another's code is correct, the verdict typically comes back crisp and final—complete with reasoning that sounds authoritative. The problem: that confidence is often unwarranted. A new paper examines why these AI judges fail to distinguish between candidates and proposes a way to know when their answers are actually justified.
The confidence-without-grounding problem#
Large language models asked to judge code quality don't naturally say "I can't tell." Instead, they produce plausible-sounding verdicts with attached explanations, making it impossible to distinguish genuine judgment from elaborate guessing. This is a critical flaw in systems designed to automate code review or verification at scale.
Multi-agent verification approaches attempt to address this by decomposing judgments into smaller, checkable claims and validating each against available evidence. This strategy has shown promise—particularly in document retrieval tasks where evidence sets naturally vary between different candidates.
Why code judging breaks the model#
The researchers identify two critical requirements for evidence-based judgment:
- Independence: The evidence must come from outside the answer being evaluated, not from reasoning about the answer itself.
- Differentiation: The evidence must meaningfully distinguish between the candidates being compared.
With traditional retrieved documents, the second condition holds by default: different search queries yield different results. Code judging, however, lacks this natural variation. When both code samples are evaluated against the same codebase or documentation, the evidence remains largely identical for both.
Testing the hypothesis at scale#
The team ran MARCH, an existing multi-agent verification framework, across 80 measurements spanning two code-judging benchmarks without modification. The results were stark: the system declared both solutions equally good in 78–95% of comparisons. Its accuracy bottomed out at 4.4%, a dramatic collapse compared to 43.7% accuracy when the same underlying model was asked to judge directly.
Remarkably, neither simpler problems nor a larger model improved this failure mode, suggesting the problem is structural rather than contingent on model capability.
