LLMs Hide Better Understanding Than They Admit—But Still Fail at Strategy

A new study from arXiv has identified why large language models struggle in strategic decision-making tasks—even when they appear to understand the situation. The gap doesn't lie in raw comprehension, but in the broken chain connecting what LLMs actually know internally to what they say and do.

Two Fundamental Disconnects#

Researchers investigating models including Llama 3.1, Qwen3, and gpt-oss discovered two critical failures in how LLMs handle negotiation and policy decisions under incomplete information:

• The observation-belief gap: LLMs construct surprisingly accurate internal representations of hidden game states—yet they tell you something quite different. When researchers examined the models' internal activations (which they call "internal beliefs" using game theory terminology), these were substantially more precise than the verbal explanations the models generated. The problem: these internal models are fragile. They degrade when reasoning requires multiple logical steps, they anchor too heavily to initial information (primacy bias) or recent updates (recency bias), and they drift further from mathematically coherent probability estimates as interactions continue.

• The belief-action gap: Even when internal beliefs are decoded accurately, the models don't reliably convert them into good decisions. Internally held beliefs actually drive weaker action choices than beliefs the model states out loud—but neither approach consistently maximizes game payoffs. The research suggests that if the models simply acted optimally based on their decoded beliefs, they'd improve outcomes in about 95% of the games tested.

Why This Matters#

These findings reveal systematic vulnerabilities in deploying LLMs for high-stakes strategic domains like real negotiations or governance without safeguards. The models aren't failing because they lack intelligence; they're failing because their internal reasoning doesn't reliably propagate to action. This makes them unpredictable and potentially unreliable partners in situations where consistent, belief-driven decision-making is essential.