AI
New benchmark reveals how LLMs exhibit cultural biases against Asian languages and cultures
Researchers introduce Camellia, a dataset measuring whether multilingual AI models favor Western entities over Asian ones across nine languages.
Researchers have created a new testing framework to measure how large language models (LLMs) handle cultural references and entities across Asian languages, uncovering systematic biases that favor Western associations over Asian ones.
The Problem: Unmeasured Bias in Multilingual AI#
Prior research has documented that LLMs tend to privilege Western-affiliated entities when processing Arabic text. However, there has been little systematic investigation into whether similar cultural preferences appear across other non-Western languages. The absence of multilingual, entity-focused benchmarks has made it difficult to assess the scope of this problem.
Introducing Camellia: A Multilingual Cultural Bias Benchmark#
A team led by Tarek Naous has developed Camellia, a benchmark designed to evaluate how well LLMs understand and adapt to culturally diverse entities. The dataset spans nine Asian languages representing six distinct Asian cultures, and includes:
- 19,530 manually labeled entities tied to either Asian or Western cultural contexts
- 2,173 masked text passages sourced from real social media, providing realistic usage scenarios
Key Findings Across Three Evaluation Tasks#
The researchers tested four recent multilingual LLMs on three distinct challenges:
- Cultural context adaptation: Whether models can correctly interpret entities within their proper cultural framework
- Sentiment association: How models link different cultures to positive or negative emotional responses
- Entity extraction in context: Whether models can accurately identify and extract entities from culturally diverse passages
Results revealed that all tested models struggled with cultural adaptation across these languages. Additionally, different LLM families showed distinct patterns of bias—some models linked particular cultures more strongly to certain sentiments than others—suggesting that biases are not universal but vary by model training and design.
