LLM surprisal alone doesn't explain how readers stumble on garden-path sentences

Language models can predict when readers will slow down on confusing sentences, but the relationship isn't as straightforward as researchers once thought. A fresh analysis of how people process garden-path sentences—temporarily ambiguous constructions that lead readers initially astray—reveals that surprisal alone (the information-theoretic measure of unexpectedness used by LLMs) is necessary but incomplete for understanding human reading behavior.

What are garden-path sentences?#

These are sentences where an initial grammatical interpretation turns out to be incorrect, forcing readers to backtrack and reanalyze. The classic example: "While the team trained the striker wondered..." Initially, readers likely parse this as the team training the striker, when in fact the team was doing the training, and the striker was wondering.

A unified framework across reading experiments#

To untangle how readers actually handle this confusion, researchers built a latent-process multinomial processing tree (MPT) model trained on data from four different experimental paradigms: eye tracking, self-paced reading (both uni- and bidirectional), and Maze. By unifying results across these methodologically distinct approaches, the model separates out three key factors:

• Likelihood of adopting a wrong initial parse
• Cost of encountering information that contradicts that parse
• Cost of syntactic reanalysis (the backtracking process itself)

Surprisal helps, but incompletely#

When the researchers compared predictions, the surprisal-only model performed worse than the full MPT architecture. Adding surprisal as an additional predictor within the MPT framework improved predictive accuracy further, suggesting that surprisal captures something real about reading difficulty—but leaves important dynamics unaccounted for. Crucially, the model also factored in inattentive reading trials, which produced more realistic estimates of the true cognitive costs involved.

Why it matters#

This work highlights a gap between how language models represent linguistic complexity and how human readers actually navigate it. Real comprehension involves not just noticing something is unexpected (surprisal), but also managing competing parse trees, detecting incompatibilities, and repairing misinterpretations. Future work on using LLMs to model human language processing will need to grapple with these additional layers.