Large Language Models Fail at Generating True Random Sequences, Amplifying Human Cognitive Biases

A new study from arXiv reveals that contemporary large language models struggle to generate genuinely random sequences, instead reflecting and often magnifying the same biases humans exhibit when making random judgments.

What the researchers tested#

Scientists examined how LLMs produce binary random sequences—specifically, simulated coin flips—drawing on classic behavioral-science methods. They analyzed model outputs across multiple dimensions: individual flips, 20-flip sequences, statistical patterns, run lengths, alternation frequencies, and predictability of subsequent flips. Results were then compared both to mathematically true random (Bernoulli) baselines and to documented human behavior from prior research.

Key findings#

The investigation uncovered that LLMs mirror several well-known human randomness biases. These include:

• Over-alternation: An excessive tendency to switch between heads and tails rather than allowing natural runs
• Run aversion: A reluctance to accept long consecutive sequences of the same outcome
• First-flip effects: Systematic patterns based on initial results

Notably, the models often didn't just replicate these biases—they amplified them significantly or introduced their own model-specific distortions. Boosting the temperature parameter reduced some rigid patterns but failed to eliminate the underlying systematic structure.

Investigating the cause#

Researchers probed why over-alternation persists through multiple angles: testing different prompts, examining continuation patterns, searching training corpora, and analyzing the models' internal computation streams. The evidence suggests the bias stems from something deeper than simple memorization or how tokens are broken down—hinting at learned patterns within the models' architecture itself.

Why this matters#

These findings carry important implications for any application relying on LLMs to handle randomness, perform fair sampling, or accurately simulate human decision-making. The results demonstrate that current models are neither reliable sources of true randomness nor faithful simulators of human behavior in this domain.