AI
Teaching LLMs to Negotiate Like Humans: Theory of Mind Reasoning Boosts Fair Deals
A study of 2,700 simulations shows that adding perspective-taking reasoning helps language models make more human-like and cooperative decisions in ultimatum games.
Researchers have found that equipping large language models with theory-of-mind (ToM) reasoning—the ability to model others' beliefs and intentions—significantly improves their capacity to negotiate fairly and align their behavior with human norms. The discovery comes from a systematic evaluation of how different reasoning approaches and value systems shape LLM decision-making in high-stakes bargaining scenarios.
The Ultimatum Game Experiment#
The team tested multiple language models, including specialized reasoning variants like o3-mini and DeepSeek-R1 Distilled Qwen 32B, across 2,700 simulations of the ultimatum game—a classic economic task where one player proposes how to split a sum of money and the other decides to accept or reject it. Before each negotiation, agents were initialized with one of three prosocial value systems: Greedy (self-interested), Fair (balanced), or Selfless (altruistic).
How Theory of Mind Makes a Difference#
Agents employing ToM reasoning—where they explicitly consider what the other player might think and want—outperformed those using standard chain-of-thought prompting on several fronts. ToM-enhanced agents achieved better negotiation outcomes, demonstrated more consistent decision-making aligned with their stated values, and behaved in ways more closely mirroring human bargaining strategies. Notably, different game roles benefited from different depths of recursive perspective-taking, suggesting that the level of reasoning required varies by context.
Key Behavioral Findings#
The research revealed nuanced patterns across agent types. Fair-minded proposers and responders who accepted reasonable offers showed the strongest alignment between their reasoning and their actions. By contrast, all agents displayed remarkable consistency with human rejection behavior when faced with unfair terms—except when the offer was genuinely equitable, suggesting a blind spot in fairness recognition.
Testing with human evaluators highlighted another insight: Llama 3.3 70B produced reasoning that most transparently matched both its observed actions and stated beliefs, signaling that reasoning quality alone matters less than coherence between reasoning and output.
