Language Models Perform Better on Some Questions When Queried in Non-English Languages

A new study reveals that multilingual language models don't treat all languages equally—some have better performance on specific topics when queried in particular non-English languages, even low-resource ones.

The Language Specificity Problem#

The finding challenges a common assumption in how these models are built. While multilingual language models are typically designed to represent semantically equivalent content across different languages in a shared conceptual space, researchers show that this approach masks an important pattern: asking the same question in different languages can yield markedly different accuracy levels.

The team introduced the concept of Language Specific Knowledge (LSK) to describe queries that perform best when posed in an "expert language" for a given model. Rather than English serving as a universal optimal input language, results suggest that different language models have specialized strengths in particular languages.

Findings Across Model Families#

The research tested this hypothesis using three datasets covering cultural and social behavioral norms. The results revealed counterintuitive language-knowledge mappings:

• Gemma models demonstrated superior performance on questions about China and the Middle East when prompted in Spanish
• Qwen models answered questions about authority and responsibility more accurately when queried in Arabic and Chinese

These patterns indicate that a model's training data distribution, regional focus, or language-specific instruction tuning creates variable expertise across different topics and languages.

The Language Selection Problem#

Researchers framed this as a new challenge: language selection—for any given query, identifying which language will yield the best model performance. To address this, they developed several baselines, including a method called LSKExtractor, designed to systematically identify optimal query languages.

Implications for Inclusive AI#

The work suggests that deploying language models more effectively requires matching queries to the languages in which those models perform best, rather than defaulting to English. This insight has practical implications for building AI systems that are more culturally aligned and responsive to diverse linguistic contexts where they're deployed, potentially improving performance across multilingual applications.