Why LLMs Struggle With Cross-Lingual Knowledge: New Research Points to Representation Unification

Large language models frequently mishandle knowledge when switching languages. When prompted about a fact in a language different from the one used during training, these systems may generate false or fabricated information—a problem researchers have long struggled to understand.

Experiments on Synthetic Multilingual Data#

Researchers set out to isolate the mechanisms behind cross-lingual knowledge transfer by training small Transformer models from scratch on controlled synthetic datasets. This laboratory approach allowed them to vary specific conditions and observe how they shaped the models' internal learning processes, avoiding the complexity of working with massive, real-world training data.

The team found that two factors determine whether models will successfully share knowledge across languages:

• Informativeness: whether facts are reliably tied to a particular language in the training data
• Extractability: how easily the model can identify which language a given input is in

The Unification Hypothesis#

The critical insight emerged from comparing different training scenarios. When both informativeness and extractability were high, models developed what the researchers call unified representations—shared internal structures that treat information the same way regardless of language. Under these conditions, factual knowledge transfers smoothly across languages.

Conversely, when these conditions weren't met, models built separate representations for each language. These isolated internal systems keep language-specific knowledge siloed, preventing cross-lingual generalization and triggering hallucinations when the model tries to answer questions outside its linguistic compartments.

Toward Better Multilingual Learning#

The framework proposed in this work reconciles several previous findings about multilingual LLMs and suggests a path forward. By understanding which training conditions encourage unified representations, researchers can design interventions—modifications to the training process itself—that would help large language models integrate knowledge across languages more reliably.

This controlled experimental approach demonstrates how simplified setups can illuminate the pre-training dynamics of real-world systems, offering practical guidance for improving cross-lingual transfer in production models.