Psychologists Need Stricter Standards for Testing Large Language Models as Research Tools

Large language models are increasingly being deployed in psychological research—both as experimental instruments and as subjects of study. But a significant methodological gap exists: many researchers apply traditional psychological tests to these AI systems without first verifying whether the results are meaningfully interpretable or trustworthy, potentially leading to what researchers call measurement phantoms—patterns that look statistically significant but don't actually measure what scientists believe they do.

The Core Problem#

The challenge arises from a fundamental mismatch in expectations. When a standard psychological instrument produces a numerical score from an LLM's outputs, researchers often assume that score has the same psychological meaning it would for a human. This assumption is frequently unwarranted. Without establishing construct validity—evidence that a measure truly captures the intended psychological phenomenon—researchers risk drawing conclusions based on statistical noise or artifacts of how language models process text.

A Framework for Responsible AI Psychology Research#

To address this gap, a new dual-validity framework proposes integrating two separate methodological traditions. The first draws from psychometrics: the study of measurement reliability and what scores actually represent. The second comes from causal inference: establishing what conclusions research findings can legitimately support. Together, these approaches create graduated evidentiary standards that scale with research ambition.

• Classification tasks (like categorizing text) require primarily accuracy and reliability—the minimal bar.
• Behavioral characterization (describing how an LLM responds) demands stronger validation.
• Human simulation (claiming an LLM exhibits traits like anxiety) requires construct validity evidence and comparison controls.
• Cognitive modeling (using LLMs to understand human thinking) sets the highest bar, necessitating experimental rigor comparable to neuroscience research.

Why It Matters#

The stakes are significant. Flawed methodology risks cluttering the scientific literature with false positives and misleading conclusions about AI capabilities and limitations. More constructively, the framework calls for developing of psychological constructs—creating new measures tailored to how language models actually work rather than simply importing human psychology wholesale into AI research.