Simpler Models Beat Transformers for Parsing Low-Resource Languages

When it comes to dependency parsing—a natural language task that identifies grammatical relationships between words—transformer models have become the go-to choice for most applications. But a fresh look at this assumption reveals a more nuanced picture, particularly for languages where annotated training data is hard to come by.

The Research#

Researchers evaluated four different parsing architectures across twelve typologically diverse languages, paying special attention to low-resource African languages. The models tested included the Biaffine LSTM (a simpler recurrent architecture), the Stack-Pointer Network, and two transformer variants—AfroXLMR-large and RemBERT. The aim was to understand how each approach performs as the availability of training data varies.

Key Finding: Simplicity Wins When Data Is Scarce#

The most striking result centers on the Biaffine LSTM, a relatively lightweight model that consistently outperformed its transformer counterparts in low-resource settings. As training data became more abundant, however, transformers gradually recovered their advantage. The crossover point—where transformers began to win—falls squarely within the range of data typical for under-resourced language treebanks.

This matters because it challenges a common assumption: that pre-trained transformer models are inherently superior regardless of context. Instead, the research suggests that simpler architectures may be more sample-efficient early on, while transformers eventually leverage their greater representational capacity once enough data accumulates.

Why Complexity Matters#

Beyond corpus size, the analysis identified morphological complexity as a significant factor shaping model performance. Languages with richer morphological systems (measured using MATTR, a lexical diversity metric) tended to show greater disadvantages for transformers relative to simpler models, even after accounting for training data volume. This hints that transformers struggle more with morphologically intricate languages when data is limited.

Implications for Language Technology#

The findings have practical implications for developers building parsing tools for under-resourced communities. Rather than immediately reaching for state-of-the-art transformer models, teams working with limited annotated data may achieve better results by starting with the Biaffine LSTM. Only once sufficient training material becomes available would a migration to transformers prove beneficial. This approach could accelerate tool development for African languages and other data-scarce linguistic communities.