Study reveals why LLMs fail to select correct answers they can internally represent

New research into language model failures reveals a puzzling phenomenon: large language models frequently produce incorrect answers despite having access to the right information buried in their internal computations. The gap between what a model can decode from its hidden states and what it actually outputs points to a fundamental issue in how these systems select their final answers.

The core problem: decodability vs. selection#

Researchers distinguish between two capabilities that don't always align. Read tests whether the correct token can be recovered from intermediate layers using appropriate controls. Write measures whether the model's final output layer ranks the correct answer first. The study found that a substantial portion of errors remain "readable"—the correct answer is mathematically recoverable from hidden states—yet the model still ranks a different token higher when generating its response.

How the final readout shapes output#

The selection gap comes down to what researchers call the selection margin: the difference in output strength between the model's chosen answer and its strongest competing candidate. This margin combines two elements:

• A context-averaged baseline tied to token frequency across the training data
• An item-specific component that varies by particular fact and context

By artificially boosting the correct answer's support to levels typical of successful generations, researchers recovered first-token selection in the majority of failures across most models studied. However, in cases where this adjustment failed, the original alternative token still dominated due to how the model's output geometry is structured.

Transfer and limitations#

Interestingly, when researchers used prompt variants that successfully generated the correct answer, the supporting evidence transferred to failing variants partly through the residual stream and through layers in the model's feed-forward network—though attention layers showed less consistent transfer.

Critically, recovering just the first token doesn't fix the full answer. Most responses remained incorrect even after boosting initial token selection, demonstrating that this intervention addresses only one layer of the problem and revealing the distinction between (can it be extracted?), (can we make the model select it?), and (does it produce coherent output?).