New Framework Distinguishes Model Confidence From Actual Capability in Large Language Models

Knowing how confident an LLM should be about its answers is crucial for safe and reliable deployment. Yet most existing calibration research has focused on a narrow angle: evaluating how correct a model's single generated response is likely to be. A new paper on the arXiv challenges this approach, arguing that it misses the bigger picture.

The Core Problem#

The fundamental issue lies in how modern LLMs work. Because these models generate responses stochastically—meaning they can produce different outputs for the same prompt each time—measuring confidence in a single response doesn't actually tell you whether the model can solve the task overall. If you ask an LLM the same question multiple times, it might get it right sometimes and wrong other times. Focusing only on one attempt obscures the model's true underlying capability.

Capability vs. Response Calibration#

Researchers introduce capability calibration (CC) as a distinct concept from the traditional response calibration (RC). Rather than asking "Is this specific output correct?", capability calibration asks "What is the probability this model will eventually solve this query?" The two frameworks differ both in theory and in real-world performance, with capability calibration proving more aligned with what practitioners actually need to know.

Why It Matters#

This distinction becomes vital in practical scenarios. When teams use strategies like pass@k—where a model attempts a task multiple times and succeeds if any attempt works—or when allocating limited inference budgets, capability-level confidence is what should guide decisions. The paper demonstrates that capability calibration better serves these applications than traditional response-level approaches.

What Comes Next#

The researchers evaluate existing confidence estimation methods to assess which approaches can feasibly achieve capability calibration, offering a starting point for future work in this direction.