When LLMs Judge Other LLMs: Why Expert Verification Still Matters

The appeal of letting one language model evaluate another is obvious: it's cheap, fast, and requires no human effort. But a new investigation into the popular "LLM-as-a-Judge" evaluation technique reveals a fundamental limitation that complicates this rosy picture.

The Problem With Ungrounded Judges#

Researchers studying how well AI systems can assess financial and business knowledge discovered that LLM judges fail in a predictable way: when judging answers on topics they themselves cannot solve, they struggle to distinguish good responses from poor ones. This matters because real-world evaluation often involves expert-level questions where correctness is more important than writing style.

The gap emerged through a specially constructed dataset called BFF-Bench, featuring 160 difficult questions and 1,200 expert-evaluated answers in finance and business. A team of financial professionals manually graded all responses, creating a ground-truth benchmark called VERDICTS. When researchers compared these human judgments against automated grading methods—including several LLM-based approaches—a clear pattern surfaced.

When Judges Can't Know the Answer#

LLM judges performed better than simpler automated methods, yet their performance hinged on a crucial detail: judges showed strong agreement with human experts only when answering questions correctly themselves. Without access to a reference answer, they could not reliably identify whether other models' responses were accurate. This blind spot meant they might praise a plausible-sounding but incorrect answer, or dismiss a correct one.

The Fix: Adding Expert References#

The research points toward a practical solution. When researchers provided the judges with expert-written reference answers, this limitation largely vanished. The judges could then effectively distinguish correct from incorrect responses across domains—even those beyond their native reasoning abilities.

Why It Matters#

As organizations deploy LLMs in high-stakes domains like finance, business, and healthcare, the need for reliable evaluation becomes urgent. This work suggests that purely automated evaluation systems carry hidden costs: they may appear to work on surface-level metrics while silently accumulating errors in specialized domains. The tradeoff between low-cost, hands-free evaluation and genuine correctness is harder to strike than it first appears.