AI
Controlled Study Reveals Pretrained Language Model Choice Drives Most Gains in Coreference Resolution
When researchers isolate the effect of the language model from task-specific design, older encoder-based systems still outperform newer decoder approaches.
A systematic reevaluation of coreference resolution models has exposed a hidden factor behind reported performance improvements in the field: much of the gain attributed to newer methods over the past five years actually stems from using larger and more capable pretrained language models, rather than architectural innovations.
The Problem: Confounded Variables#
Coreference resolution—the task of identifying which words and phrases refer to the same entity in a text—sits at the heart of many language understanding systems. However, researchers face a methodological challenge: state-of-the-art models build atop pretrained language models like BERT or larger variants, making it nearly impossible to isolate whether one model genuinely performs better due to its design or simply because it leverages a more powerful underlying language foundation.
What the Study Found#
By controlling for the pretrained language model and testing five coreference resolution models under identical conditions, researchers discovered several surprises. Encoder-based architectures (the older approach) outperform more recent decoder-based models on both accuracy metrics and inference speed. More strikingly, newer encoder models do not consistently achieve higher accuracy than their older counterparts—and the oldest model tested actually generalized best when evaluated on text from different domains and genres.
Rethinking Model Progress#
When the researchers accounted for language model selection, most of the F1 score improvements announced over a five-year period vanished. This suggests that published advances in coreference resolution have been driven largely by access to better pretrained foundations rather than algorithmic breakthroughs specific to the task itself.
The findings highlight the importance of fair comparison in machine learning research and raise questions about which improvements genuinely represent methodological progress versus those that simply reflect the availability of larger, better-trained models.
