AI models secretly know when they're wrong—but can't admit it

Researchers trained a 318-million-parameter language model on pure Classical Chinese and discovered a curious phenomenon: the model's internal representations showed clear signs of knowing the difference between real and invented information, but its actual output revealed no such awareness.

The internal knowledge#

When tested with out-of-distribution inputs, the model displayed a sharp rise in perplexity—a key measure of how confused an AI is—when encountering false information. Real historical events produced a baseline perplexity, while entirely fabricated ones triggered a 2.39-fold jump. Semi-fabricated scenarios (real historical figures placed in made-up situations) triggered an even starker 4.24-fold increase. These differences were statistically significant and suggested the model had genuinely encoded factual knowledge beyond mere pattern recognition.

The expression problem#

Yet when the researchers examined what the model actually produced, this internal knowledge vanished. The model failed to use epistemic markers—the Classical Chinese equivalent of phrases like "I think" or "perhaps"—when encountering out-of-distribution queries. In fact, it used such hedging language less frequently for uncertain cases (3.5%) than for confident ones (8.3%). This suggested the model was simply mimicking training data conventions rather than expressing any genuine understanding of its own limitations.

Broader patterns across languages#

The researchers replicated their findings across three languages—Classical Chinese, English, and Japanese—across multiple writing systems, and across eight models ranging from 110 million to 1.56 billion parameters. Interestingly, different training datasets produced wildly different behaviors. Classical Chinese models showed a "humility paradox," hedging more when discussing well-known topics. Japanese models, by contrast, almost never used uncertainty markers at all, regardless of the task.

Why it matters#

The work suggests that learning to express doubt—what the researchers call "metacognitive expression"—does not emerge naturally from language modeling. Instead, it appears to require explicit training through techniques like reinforcement learning from human feedback (RLHF). The implication is clear: a model may privately know it's wrong, but without special intervention, it will confidently state falsehoods anyway.