New Benchmark Reveals How LLMs Fail to Align with Values Across Different Cultures and Regions

Researchers have created a comprehensive evaluation framework that uncovers significant gaps in how large language models align with values across the world. The new benchmark, DiverValue-Bench, tests AI systems against the priorities and perspectives of people in 74 countries and regions, revealing performance discrepancies that standard benchmarks typically miss.

A Global Approach to Value Alignment#

Most existing assessment tools treat LLM performance as a single aggregate score, masking the reality that what matters to users varies dramatically by geography, culture, and demographics. DiverValue-Bench takes a different approach by treating value alignment as inherently population-specific. The benchmark comprises nearly 24,000 carefully vetted test instances sourced from the PRISM user feedback dataset, with each example labeled with fine-grained value dimensions and paired with personalized questions, contrastive answer options, and detailed metadata about respondents' backgrounds.

The research team validated this data through large-scale human review to ensure quality and relevance across diverse populations.

Exposing Hidden Disparities#

When the researchers evaluated representative LLMs using DiverValue-Bench, they found that standard performance metrics obscure substantial geographic and demographic variation. Some models performed well on average while failing systematically for certain regions or demographic groups—a pattern invisible when results are aggregated. These disparities underscore why alignment evaluation must move beyond one-size-fits-all metrics.

Improving Alignment Through Targeted Fine-Tuning#

The team demonstrated that preference-based fine-tuning approaches can meaningfully improve value alignment. Using lightweight techniques including Low-Rank Adaptation (LoRA) and Direct Preference Optimization (DPO), they showed that models could be adjusted to better reflect diverse values. Importantly, gains achieved on the benchmark also transferred to out-of-domain scenarios, suggesting the improvements are robust rather than narrow overfitting.

Why This Matters#

The work highlights a fundamental gap in AI development: current practices often fail to account for the diversity of human values that should guide system behavior. By providing a practical, large-scale tool for population-aware alignment evaluation and training, DiverValue-Bench offers a foundation for building AI systems that work equitably across global populations rather than serving narrow, homogeneous preferences.