New framework improves how LLMs judge text quality while catching errors

Large language models are being deployed at scale to compare and rank text outputs—a task called pairwise evaluation. But these AI judges suffer from built-in miscalibration and hidden biases that can skew results. Researchers have introduced SCOPE (Selective Conformal Optimized Pairwise Evaluation), a new approach designed to make LLM-based judging more trustworthy by calibrating the decision threshold so that errors stay within a target risk budget.

The calibration problem#

Using LLMs as judges is appealing because it's cheap and fast compared to hiring humans. Yet the models don't always know when they're uncertain, and they can favor certain response orders (phrasing the same comparison differently can flip the result). Existing confidence measures—standard probability scores and similar proxies—often fail to catch these failure modes, leaving evaluators blind to when a judgment might be wrong.

Bidirectional preference entropy#

To give SCOPE a better uncertainty signal, the team introduced Bidirectional Preference Entropy (BPE), a scoring method that queries the judge twice: once with each response in the first position. The method converts these order-dependent preference probabilities into an entropy score that doesn't favor either side of the comparison. Early testing showed BPE outperforms standard confidence measures in both calibration (staying true to its stated error rate) and discrimination (separating strong judgments from weak ones).

Guarantees under uncertainty#

SCOPE pairs BPE with conformal prediction, a statistical framework that guarantees error rates under mild assumptions about data exchangeability. Users set a target error rate (for example, alpha = 0.10 means at most 10% errors), and the system automatically finds a threshold. Any judgments below that threshold get flagged for human review; those above it are trusted. Across multiple benchmarks, SCOPE reliably achieved its target error rate (empirical false discovery rate around 9.7–9.9% when targeting 10%), while also retaining most judgments for automated use—up to 2.4 times more coverage than simpler baselines operating under the same error constraint.

Implications for evaluation at scale#

The method suggests a practical path forward for organizations using LLMs to run large evaluation campaigns. Rather than assuming every model judgment is reliable, SCOPE provides a statistical guarantee that accepted judgments meet a quality threshold, with uncertain cases escalated to human review.