New benchmark tests whether AI can predict individual human reasoning, not just average opinions

Large language models have become remarkably skilled at mimicking human responses, yet they typically converge toward population-level consensus rather than capturing the distinct reasoning patterns of individuals. Researchers have now created a new benchmark to explore whether AI can move beyond this limitation and predict how a particular person would think and update their beliefs when confronted with new information.

A shift from average to individual#

HugAgent addresses a fundamental gap in how we evaluate AI reasoning. Rather than testing whether models can approximate the typical human response to a question, the benchmark asks whether they can replicate the thinking of a specific person—including their prior beliefs, their reasoning style, and most importantly, how they would adjust their views in response to intervention. This reframing acknowledges that genuine human-like reasoning involves not just holding beliefs but evolving them in response to evidence and argument.

How the benchmark works#

The research team collected data from human participants across three policy domains—healthcare, surveillance, and zoning—using a combination of structured questionnaires and semi-structured interviews where participants explained their thinking aloud. This approach captures what researchers call "ecologically valid belief states": genuine, context-grounded reasoning rather than idealized responses to hypothetical scenarios.

Models are evaluated on two fronts:

• Belief state prediction: Can the model infer a person's current views from partial information about their prior stance?
• Belief update prediction: Can the model predict how that same person's reasoning would shift when exposed to new arguments or data?

Where models fall short#

The experiments reveal a striking asymmetry. When given evidence of what someone currently believes, AI models can reasonably reconstruct that belief state. But they struggle significantly when asked to predict how that person would respond to change. Rather than modeling identity-consistent reasoning—the continuity of a person's underlying reasoning approach—models appear to rely on surface-level topical associations, matching patterns within a domain rather than understanding the deeper logical coherence of how an individual thinks.

The findings suggest that simply providing models with more context won't close this gap. Instead, progress requires better mechanisms for detecting and predicting genuine cognitive shifts—the ability to recognize when and how a mind changes, not just what it currently thinks.