Study Reveals LLMs Struggle to Gauge Their Own Accuracy More Than Previously Thought

A new research study from arXiv challenges conventional methods for evaluating how well large language models understand the limits of their own knowledge. Using signal detection theory—a framework borrowed from perceptual psychology—researchers found that standard confidence metrics mask important gaps between what LLMs know and how well they recognise what they know.

The Problem with Current Evaluation Methods#

Most approaches to measuring LLM confidence rely on calibration metrics such as Expected Calibration Error (ECE) and Brier score. These measures, however, blend two distinct capabilities: how accurate a model actually is at a task, and how effectively its expressed confidence reflects that underlying accuracy. By conflating these dimensions, researchers say, we miss critical insights into model behaviour.

A New Lens: Signal Detection Theory#

The team applied signal detection theory (SDT), traditionally used in psychology and neuroscience to study perception, to decompose these two capacities. They treated the model's token-level probability output as a graded confidence measure and correctness as the signal to be detected. This approach allowed them to isolate and measure what researchers call Type-2 metacognitive sensitivity—essentially, how well the model's confidence tracks its actual performance.

Key Findings Across Multiple Models#

Testing four models—Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.3, Llama-3-8B-Base, and Gemma-2-9B-Instruct—across roughly 224,000 factual question-answering trials, the researchers uncovered several notable patterns:

• Metacognitive information varies sharply: Different models show more than two-fold differences in how informative their confidence signals are, with less accurate models sometimes showing the most informative confidence ratings
• Hidden structural differences: Confidence signals have model-specific variance structures invisible to traditional calibration metrics, with z-ROC slopes ranging from 0.81 to 1.18
• Domain-specific variation: Metacognitive efficiency is strongest in Arts & Literature across all tested models
• Temperature decouples confidence from accuracy: Adjusting temperature shifts accuracy while leaving the confidence signal relatively stable