New method helps AI models give more honest confidence estimates about their answers

Large language models frequently voice confidence levels that don't match whether their answers are actually correct—a problem that threatens their usefulness in situations where reliability matters. Researchers have now introduced a method that addresses this gap by letting models learn from their own experience.

The Problem with Overconfident AI#

When an LLM responds to a question, it can sound extremely certain even when it's wrong. This mismatch between stated confidence and actual accuracy makes these systems risky for applications like medical diagnosis or legal advice. Earlier approaches to fixing this issue treated it as a one-time puzzle at inference, either through careful prompting or by asking the model to second-guess itself after generating an answer.

How Rehearse Works#

The new approach, called Rehearse (Experiential Rehearsal), takes a different path. Rather than trying to fix calibration in a single exchange, it enables models to adjust based on patterns in their own past performance.

The process unfolds in two phases:

• Learning phase: The model plays a confidence-calibration game where it makes predictions and receives scored feedback on whether its confidence matched the actual outcomes. This experience gets summarized into a "trajectory prefix"—essentially a memory of whether the model has been systematically overconfident or underconfident.

• Application phase: When answering new questions, the model draws on this cross-instance calibration signal as it reasons through the chain of thought, naturally tempering or boosting its stated confidence based on learned patterns.

Importantly, this requires no additional training, only feedback from a properly designed scoring rule.

Results Across Multiple Tests#

Testing the method on four different language models using three separate benchmarks showed consistent gains. The researchers measured improvement using Expected Calibration Error (ECE), a standard metric for confidence accuracy. Rehearse achieved the lowest average ECE among training-free competitors and reduced it by 58% compared to uncalibrated baselines—while also improving raw accuracy.

The code has been made publicly available for other researchers to build on the findings.