Language Models Hide Knowledge of Falsehoods in Low-Dimensional Geometric Structures

Language models often produce plausible-sounding false statements as readily as true ones, leaving researchers unable to determine from surface output whether the model "knows" a claim is wrong. A new paper published on arXiv reveals that signals about correctness do exist in the models' internal mechanics—and their structure is far simpler than expected.

A Hidden Geometric Pattern#

Researchers examined internal representations across 11 different language models (ranging from 124 million to 14 billion parameters) and discovered that truth-related information occupies a strikingly compact space. Rather than being scattered across hundreds of dimensions, the signal concentrates in just 2–8 dimensions, where correct and incorrect statements cluster into distinct groups, much like points separated into classes in a 2D plot.

This organization is so regular that a simple linear detector—the geometric equivalent of drawing a line through the data to separate the two categories—performs nearly as well as more complex trained models. Remarkably, just 25 labeled examples were sufficient to recover 90% of the performance achieved with full training data on GPT-2.

Testing the Geometry Directly#

The team validated these findings through three causal intervention methods:

• Activation steering: Deliberately shifting internal activations along the identified dimensions changed hallucination rates by up to 9.1 percentage points across six models.
• Concept erasure: Removing the signal entirely dropped detection accuracy to chance levels, confirming its necessity.
• Distributed alignment search: This ranking-aware technique located at most five causal dimensions, the tightest bound reported.

Practical Limits and Cross-Model Transfer#

The internal representations proved more effective than standard confidence measures like model probability or semantic entropy, particularly for detecting adversarial misconceptions. However, this advantage narrows when tested on conventional QA benchmarks.

A major limitation emerged in transfer learning: probes trained on a single dataset transferred poorly to others, suggesting the representations are dataset-dependent. The solution was joint multi-dataset training, which restored performance to 73–91% accuracy across models, hinting that better training strategies could improve generalization.