LLM Digital Twins Show Promise but Struggle to Match Human Psychology Across Tasks

Large language models are increasingly being used as "digital twins"—simulated versions of human respondents for research and testing—but a comprehensive study raises important questions about how well they actually replicate human psychology.

Researchers have now evaluated whether LLM-based digital twins are psychometrically valid compared to humans, using a framework that tests both construct representation and nomothetic span (the relationships between different psychological constructs). The results paint a nuanced picture: LLMs excel at mimicking aggregate group behavior but falter when it comes to capturing individual quirks and decision-making patterns.

Where LLMs Succeed#

At the group level, digital twins performed well, achieving high accuracy when aggregating responses and showing strong correlations with human response profiles. In word association tasks, LLM networks even replicated the humanlike "small-world" structure that characterizes how concepts connect in human minds—and they organized words into theory-consistent semantic categories.

Where They Fall Short#

However, the picture darkens at finer levels of analysis. Item-level correlations were substantially weaker, meaning individual responses diverged from human patterns. In decision-making tasks, LLMs displayed normative rationality—they made logically optimal choices rather than human choices. They failed to reproduce classic cognitive biases like anchoring or framing effects, instead showing compressed variance in their responses and limited sensitivity to temporal context.

When it came to predicting personality traits like the Big Five, feature-rich conditioning (providing more detailed background information) helped improve accuracy, yet the underlying network structure remained inconsistent across different configurations, revealing persistent gaps in how the models represent personality.

Cross-Language and Qualitative Findings#

In multilingual free-text tasks conducted in English and Chinese, enriched digital twins better captured overall construct-level meaning in narratives. Yet linguistic differences and idiosyncratic variation persisted, suggesting LLMs impose their own patterns rather than fully absorbing individual human voice.

Implications for Practice#