How a Language Model's First Language Interferes with Learning a Second One

Researchers have uncovered how language models trained on one language tend to carry over its grammatical patterns when learning a second one—a phenomenon mirroring the cross-linguistic interference that shapes human bilingualism.

The Core Finding#

When language models acquire a second language after mastering a first, the structural properties of their initial training bleed through into new linguistic tasks. The strength of this crosslinguistic influence (CLI) depends on how dominant the first language was during training and how different the two languages are from each other structurally. The team found this effect is not a random byproduct of limited model capacity, but rather a predictable outcome of how neural networks organize their linguistic knowledge.

How the Study Worked#

Researchers trained artificial bilingual systems across 15 typologically diverse language pairs, varying when the second language appeared during training (called "Step of Exposure"). Using a technique called crosslinguistic structural priming, they measured both positive transfer—where the first language helps learning the second—and negative transfer, where it creates interference. The key finding: higher first-language dominance (longer exposure before introducing the second language) amplifies this structural carryover, but it also weakens the model's ability to learn genuinely different grammar patterns.

The Mechanism Inside the Network#

By analyzing the internals of these models, the researchers discovered that languages with similar grammatical structures literally share overlapping neurons in the network. When the researchers explicitly primed the models with first-language structures, grammatical processing migrated toward the deeper layers of the network. Targeted ablations revealed that only the attention mechanisms in these final layers actually drive the cross-lingual transfer—they route the first language's structural blueprint into the final prediction, overriding what the model might otherwise learn.

Why This Matters#

The study demonstrates that CLI in language models is a structured phenomenon governed by language dominance and learning proficiency, not a flaw. Understanding this mechanism could improve how to train multilingual AI systems and help researchers design models that learn new languages more robustly by better managing the influence of prior linguistic knowledge.