Researchers unlock hidden knowledge in large language models through attention head analysis

Large language models frequently store correct answers internally but fail to select them when answering multiple-choice questions. Researchers have discovered a way to extract this latent knowledge by analyzing how attention heads align query vectors with key vectors—a finding that could improve model reliability without retraining.

The Discovery: Hidden Correct Answers#

The phenomenon reveals a gap between what LLMs know and what they output. Models regularly encode the right option for multiple-choice questions (MCQA) internally, yet produce wrong answers. By examining the Query-Key (QK) score—the dot product between the last-token query and keys at option-marker positions, measured before rotary positional embeddings apply—researchers identified special select-and-copy heads in middle layers that perform option selection through semantic alignment.

These heads differ mechanistically from previously known head types. They remain invariant to label symbols and can solve synthetic tasks with zero surface overlap, properties that require stripping positional embeddings to observe. This distinction suggests a genuinely different selection mechanism than simple position-based copy operations.

Testing Across 24 Models#

The research spans models ranging from 1.5B to 72B parameters, including LLaMA-2/3/3.1/3.3, Qwen-2.5, Gemma, Phi-3.5, and DeepSeek-R1-Distill. Results show dramatic gaps between model outputs and latent accuracy. A single head's QK-score exceeded zero-shot accuracy by up to +27.4 percentage points on HellaSwag and +49.8 percentage points on HaluDialogue. Removing these heads through causal ablation collapses performance to near-random levels, confirming their criticality.

Unsupervised Head Identification#

To avoid dependency on labeled validation data, the team developed HeadScore, an unsupervised ranking method that identifies high-value heads using only unlabeled inputs. This approach recovers the supervised top-performing heads on every tested model, making the technique practical for deployment without additional annotation.

Complementary to Existing Methods#

The QK-score method works alongside—not in place of—existing positional-debiasing techniques like PriDe and Wang's approaches. Traditional debiasing methods reweight output logits after decoding, whereas QK-score reads the model's selection from within middle-layer computation. This complementary nature suggests potential for combined approaches.