AI
LLMs Can Now Auto-Generate Personality Tests—With Surprising Accuracy
A new framework shows that GPT models can reliably create situational judgment tests for personality assessment, potentially replacing labor-intensive expert-designed methods.
Personality assessment has traditionally relied on situational judgment tests (SJTs)—specialized questionnaires where respondents react to realistic scenarios—but creating them has always been a bottleneck. The process demands significant time, money, and involvement from subject matter experts. A new study from researchers at arXiv suggests that large language models like GPT-4 and ChatGPT-5 can now automate much of this work without sacrificing quality.
How the Research Unfolded#
The investigation consisted of three separate studies testing different aspects of AI-generated personality tests. The first looked at how prompt design and temperature settings (a parameter controlling how creative versus consistent an LLM's output is) affected the quality of generated items. Using GPT-4, the team discovered that carefully crafted prompts combined with a temperature of 1.0 struck the best balance between originality and accuracy.
The second study extended this approach to ChatGPT-5, testing whether the method remained stable and reproducible across multiple runs and different models. Results showed consistent, high-quality output, suggesting the framework generalizes well beyond a single system.
Testing the Real-World Validity#
The final stage evaluated whether LLM-generated tests actually measure what they claim to measure across five dimensions of the Big Five personality model. The tests demonstrated satisfactory reliability and validity on most dimensions, though some limitations emerged: the compliance facet showed weaker convergent validity, and certain connections between test scores and real-world outcomes needed improvement.
Why This Matters#
If validated further, this approach could substantially reduce the time and expertise required to build personality assessments—a win for researchers, HR professionals, and organizations. The findings demonstrate that LLMs, when prompted strategically, can generate psychometrically sound instruments suitable for diverse cultural contexts. Whether the method can scale to all assessment types and contexts remains to be seen, but the evidence so far is encouraging.
