LLMs capture social reasoning patterns but struggle with magnitude accuracy

A new study examines whether large language models don't just mimic human social reasoning qualitatively but also match it in quantitative precision. Researchers developed tailored measurement approaches to separate two distinct challenges: whether an LLM captures the structure of how humans make social inferences versus whether it gets the magnitude of those inferences right.

Measuring structure versus magnitude#

The team introduced two calibration-focused metrics to isolate these different dimensions. The Effect Size Ratio (ESR) and Calibration Deviation Score (CDS) allow researchers to distinguish between models that understand the logical relationships of social meaning and those that merely reproduce them at the wrong scale.

This distinction matters: an LLM might correctly recognize that vague language carries social signals about speaker knowledge, yet consistently overstate or understate how strong those signals are. The two new metrics make it possible to measure each failure independently.

Testing pragmatic reasoning through prompting#

The researchers tested two intervention strategies grounded in pragmatic theory. Both rest on the idea that people derive social meaning by reasoning about what speakers choose to say versus what they could have said, and by inferring what speakers actually know and intend to communicate.

The first approach prompted models to consider alternative utterances—essentially asking the model to think about what the speaker didn't say. The second encouraged explicit reasoning about speaker knowledge and motives. The third approach combined both.

Results across frontier models#

Testing on a case study involving numerical (im)precision—a domain where social meaning is measurable—the researchers examined three frontier LLMs. All three successfully replicated the qualitative structure of human social inference. They understood that imprecise numbers carry social meaning about what a speaker knows.

However, the models differed substantially on magnitude calibration. Prompting for reasoning about speaker knowledge and motives consistently reduced how far off-track the models went. Prompting for awareness of alternatives, surprisingly, tended to make exaggeration worse. Only when both components were combined did the intervention improve all calibration-sensitive metrics across all models.