New benchmark reveals video AI models struggle with cultural norms outside Western contexts

Researchers have introduced a new evaluation framework to test how well video language models understand cultural norms across different societies. The work reveals significant performance gaps, particularly when these systems encounter non-Western cultural contexts.

The VideoNorms Dataset#

The team created VideoNorms, a collection of cultural norm annotations drawn from popular television shows in the United States and China. Each annotation includes a label indicating whether a scene adheres to or violates expected cultural norms, along with evidence—both verbal and non-verbal—supporting that judgment.

The dataset contains over 3,000 human-verified judgments, assembled through a structured process: a large video language model initially labeled each scene, and then at least three trained annotators with deep cultural knowledge reviewed the results. This human-AI collaboration approach ensured that cultural assessments reflected genuine expertise rather than automated guesses alone.

Key Performance Gaps Identified#

Testing seven open-weight video language models revealed consistent weaknesses:

• Geographic disparities: Models perform substantially worse when analyzing Chinese cultural content compared to US material, with particularly sharp drops in their ability to identify adherence to cultural norms
• Modality challenges: Systems struggle more when explaining non-verbal forms of norm adherence or violation—such as gesture, body language, or visual symbolism—than when parsing spoken dialogue
• Scaling limits: Simply increasing model size does not improve classification accuracy, suggesting the problem runs deeper than model capacity

The researchers' statistical analysis, which employed hierarchical linear modeling, underscores a troubling pattern: models trained predominantly on Western data fall behind when generalizing to cultures underrepresented in their training sets.

Implications for Global Deployment#

The findings caution against deploying video understanding systems worldwide without explicit cultural grounding. The work points toward the need for training approaches that better balance representation across cultural contexts and for evaluation frameworks that systematically test cross-cultural reasoning before release.