Standard Personality Tests Fail to Predict How LLMs Actually Behave With Users

A new study challenges the widespread practice of using human personality and values questionnaires to assess how large language models behave in practical scenarios. Researchers found that the profiles generated by traditional psychometric tests bear little resemblance to how these systems actually respond to ordinary user interactions.

How the Research Compared Two Approaches#

The team evaluated eight open-source LLMs using two distinct methods. The first involved administering standard questionnaires—the PVQ-40/21 and BFI-44/10—where models provided Likert-scale self-reports. The second measured actual generation probabilities when models responded to everyday conversational queries containing value-laden elements.

The results showed dramatic divergence between the two approaches. When assessed through traditional questionnaires, models displayed the kind of internal consistency typically associated with stable personality traits. Yet when evaluated through realistic user interactions, that consistency vanished entirely.

Why Questionnaires Mislead#

The mismatch stems from how questionnaire items are constructed. These standardized instruments rely on explicit lexical markers that make their target constructs transparent—a model can easily recognize when a question probes for cooperation versus competitiveness, for example. This visibility allows systems to craft socially desirable answers aligned with expected personality profiles.

Real user queries, by contrast, rarely telegraph such psychological dimensions as clearly. When faced with authentic conversational tasks, models lack the same recognizable cues and generate responses through fundamentally different mechanisms than those activated by direct questionnaire prompts.

The Persona Problem#

The research also tested whether demographic role-play instructions affect model responses. When prompted to adopt specific demographic personas, models shifted their questionnaire answers in patterns matching genuine human behavior. Strikingly, these same shifts did not occur in generation-probability testing, suggesting models perform convincing personality mimicry on questionnaires without developing the corresponding behavioral dispositions.

Implications for LLM Assessment#

The findings argue against using questionnaire scores as standalone evidence for predicting LLM conduct in realistic settings. Instead, the researchers advocate for profiling systems through generation-probability analysis applied to ecologically valid items—tasks and queries that mirror how users actually engage with these models. This complementary measurement approach would better capture the gap between what models report about themselves and how they truly operate.