Semantic consistency beats token confidence for trustworthy AI scientific answers

When deploying large language models for scientific question answering, knowing when to trust their outputs is critical — yet pinpointing uncertainty across lengthy, multi-step reasoning remains unsolved. A new benchmark study demonstrates that semantic consistency measures vastly outperform traditional confidence-scoring methods for this high-stakes task.

The calibration problem#

Determining whether an AI system's answer is reliable is essential for real-world deployment, particularly in domains like science where human verification of full responses becomes impractical at scale. The team behind the research published on arXiv tested four uncertainty quantification methods across 685,000 responses from up to 20 different language models on seven datasets, with support from a reusable open-source framework.

How instruction tuning distorts confidence signals#

The research reveals a persistent flaw in how modern models report their own certainty. When language models undergo instruction tuning — a technique used to improve their ability to follow directives — their confidence distributions collapse into extreme values. Rather than expressing nuanced uncertainty, these models polarize their token-level probabilities, rendering their per-token confidence signals fundamentally unreliable for gauging actual accuracy.

Interestingly, reasoning models diverge in how they handle this problem: some inherit the same polarization pattern while others actively counteract it. This divergence clusters by company, pointing to deliberate differences in training pipeline design as the root cause.

Why aggregation methods fail#

Methods that attempt to combine token-level signals — whether through verbalized confidence (asking the model to state its confidence) or by aggregating probabilities across a sequence — consistently underperformed. These approaches break down precisely where they're needed most: in complex, interconnected reasoning tasks where errors in one step cascade through subsequent logic.

The winning approach: semantic consistency#

Only one method proved reliable at scale: measuring whether the model generates the same final answer when asked repeatedly. This semantic consistency approach — essentially checking internal agreement — provided well-calibrated uncertainty estimates even in multi-step, dependency-rich reasoning settings. The finding represents the first large-scale evidence that this simple consistency check can work reliably for scientific QA, offering a practical path forward for deploying AI systems where accuracy assurance matters.