Training Models to Predict When LLMs Are Right, Not Just to Trust Themselves

Deploying large language models in critical applications—medical diagnosis, legal analysis, financial decisions—hinges on knowing when to trust their answers. Yet teaching an LLM to estimate its own correctness has proven stubbornly difficult. A new paper challenges a widespread assumption: that models possess special "self-knowledge" about their own accuracy.

The Problem With Self-Awareness#

Researchers tested whether an LLM's ability to judge the correctness of its own outputs actually performs better than having an entirely different LLM make the same judgment. The results were surprising: it doesn't. This finding undermines the conventional framing of confidence estimation as primarily a problem of unlocking a model's internal understanding.

The team proposes a different approach, calling it a Generalized Correctness Model (GCM). Rather than relying on any single model's introspection, a GCM is trained to recognize patterns in how answers tend to be correct or incorrect—learned from the historical track record of many different models across multiple datasets.

How It Works#

The researchers experimented with several methods of incorporating this historical correctness information. They trained GCMs on data from five different model families and tested them on MMLU and TriviaQA benchmarks, finding that the approach generalizes well across both models and datasets. Interestingly, they discovered that how an answer is phrased emerges as a surprisingly strong predictor of whether it will be correct.

Beyond training new models, they also explored simpler alternatives: inserting historical examples directly into the model's context window, and applying post-hoc calibration techniques to refine confidence scores after the fact.

Why It Matters#

The findings suggest that reliable confidence estimation is not a specialized skill locked inside each model. Instead, it's a learnable, model-agnostic capability that arises from systematically studying patterns in past predictions. This shifts the engineering challenge from trying to get each model to introspect better, to building systems that learn from collective historical performance.

For applications like selective prediction—where the system can defer to a human when uncertain—this approach offers a clearer path to safer AI deployment.