AI
Bigger Language Models Predict Brain Activity and Reading Patterns Less Accurately Than Expected
A new study challenges the assumption that larger AI language models better capture how humans process sentences by controlling for model complexity.
Researchers testing whether larger language models (LLMs) align better with human cognition have found a counterintuitive result: bigger doesn't mean better when it comes to predicting how people read or how their brains respond to text.
The research, submitted to arXiv, challenges a widely held theory that as language models grow more powerful at predicting words, their alignment with human sentence processing should improve correspondingly. Instead, the study reveals that this scaling relationship actually reverses at larger model sizes—a phenomenon known as inverse scaling.
The Quality-Power Hypothesis Under Scrutiny#
Scientists have been intrigued by the possibility that large language models might serve as computational models of how humans parse and understand language. This idea rests on a "quality-power" relationship: the theory that LM performance on human psychometric data (like reading times and brain imaging measurements) should monotonically improve as the models' ability to predict words in context increases. If true, this would suggest that LLM architecture shares fundamental features with the human language comprehension system.
However, previous research already hinted that this neat relationship breaks down. Studies using information-theoretic surprisal—a measure of how unpredictable words are in context—showed signs of inverse scaling. Other work explored using full vectors from LLMs of various sizes, still reporting positive relationships, but without rigorous controls for the effects of simply having higher-dimensional representations.
Controlling for Dimensionality Expansion#
The key methodological innovation in this new study is its use of untrained baseline models with up to 66 billion parameters. By comparing trained LLMs against untrained models of equal size, researchers could isolate the true predictive signal learned during training from the statistical effects of having more dimensions in the vector space.
The results support the inverse scaling pattern: larger trained models actually perform worse than smaller ones at predicting human reading times and brain imaging data, once dimensionality effects are accounted for. Even more strikingly, the added value of trained LLMs over their untrained counterparts drops to nearly zero around a few billion parameters on most datasets.
