LLMs can replicate human cooperation patterns in game theory experiments

Researchers have tested whether large language models can faithfully reproduce human behavior in strategic games, finding that some models align far more closely with real human choices than others. The results carry implications for both AI safety and the use of LLMs as tools for social science research.

The Experiment#

The team replicated large-scale game-theoretic experiments using three widely deployed open-source models: Llama, Mistral, and Qwen. They ran 121 dyadic games—two-player interactions—spanning four classical game types to measure how well each model's choices matched documented human behavior patterns. Notably, they achieved this matching without resorting to persona-based prompting, which typically tries to nudge models toward specific behavioral traits.

Key Findings#

The models displayed starkly different decision patterns:

• Llama reproduced human cooperation strategies with high fidelity, suggesting the model's internal learning process may have captured real human social preferences
• Qwen aligned more closely with Nash equilibrium predictions—the theoretically optimal outcomes when each player makes rational choices—rather than mirroring what actual humans do
• Mistral occupied a middle ground, diverging from both human patterns and game-theoretic ideals

Behavioral profiling revealed that humans and Llama both displayed an "envious decision profile," meaning both showed sensitivity to how their outcomes compared to their partner's outcomes. Qwen and Mistral exhibited different profiles altogether.

Why This Matters#

The findings highlight a critical tension in deploying LLMs: models currently used as decision-making agents in important domains may not reliably replicate human reasoning. This gap matters both for safety—misalignment could produce unintended harm—and for science, where researchers increasingly use LLMs to simulate human behavior in unexplored scenarios.

An analysis of how Llama processes payoff information revealed a structured, layer-dependent approach absent in the other models, suggesting a mechanistic explanation for its closer alignment with human choices. The researchers leveraged this capability to extend experiments into new game configurations and pre-registered testable hypotheses about human behavior in scenarios never before empirically tested.

The work demonstrates that appropriately designed LLMs can serve as useful complements to traditional behavioral research, generating novel predictions about human social decision-making.