LLM Judges Ignore Authorization to Abstain From False Binary Calls on Conflicting Evidence

Large language models tasked with fact-checking are frequently making assertive judgments when they should be declining to choose a side. An arXiv paper (2606.07834) documents a systematic pattern: LLM judges routinely return a directional verdict—supports or refutes—on claims where their task contract explicitly authorizes neutral options like Conflicting or Not Enough Evidence.

What the Problem Is#

The researchers frame this as directional overcommitment: a type of label error where an LLM outputs a verdict that its own instructions forbid. When both sides of a conflict are present in evidence yet the model commits to one direction anyway, the team calls this Cherry-pick Override (CPO). Unlike cases where evidence is simply insufficient, CPO cannot be solved by gathering more information—the model is selectively ignoring material it should acknowledge.

The Evidence#

Testing showed troubling rates across multiple judges:

• A human panel of four reviewers committed directionally on 18.7% of claims they should have flagged as conflicting
• Four frozen contemporary LLM judges (two proprietary, two open-source) overcommitted on 21.3–48.0% of such claims
• Performance degraded further on VitaminC-Mixed, a constructed stress test dataset, where overcommitment reached 24.0–38.0% and rankings reversed between judges

The researchers emphasize these are conservative lower bounds: their shared prompt explicitly defines Conflicting, forbids collapsing to a direction, and demonstrates the label three times.

Broader Pattern#

The failure extends beyond conflict detection. Overcommitment on cases with genuinely insufficient evidence occurred at comparable rates across all judges and actually exceeded CPO rates for three of the four LLMs on the stress test. When asked in a fresh context to re-examine their own prior errors, two judges correctly identified the conflict roughly two-thirds of the time—indicating the knowledge was available but never reached the final decision.

Attempted Fixes Fall Short#

The team evaluated multiple interventions—typed verdict vocabulary, panel voting, confidence thresholds, structural validators, and self-decomposition—each leaving substantial residual failures. The most successful approach, self-decomposition, partly achieved its gains by relabeling one-sided evidence as conflicting rather than genuinely improving judgment quality.