New Vietnamese benchmark exposes vision-language model weaknesses beyond OCR

Researchers have unveiled VMMU, a comprehensive benchmark designed to test how well vision-language models (VLMs) can understand and reason about visual and textual information in Vietnamese. The work challenges assumptions about model capabilities and points to deeper reasoning limitations.

What VMMU measures#

The benchmark comprises 2,500 multimodal questions spanning seven distinct tasks. Questions demand genuine integration of visual and textual inputs—solving STEM problems, interpreting data, applying visual rules, and working through abstract reasoning. Crucially, the problems cannot be solved by relying solely on text extraction or optical character recognition shortcuts; models must actually combine information from both modalities.

Key findings#

When tested on VMMU, leading proprietary vision-language models achieved only 66% mean accuracy on average. This result is striking given that the same systems demonstrate strong Vietnamese optical character recognition performance. The researchers' deeper analysis revealed that the actual bottleneck is not the ability to read Vietnamese text from images, but rather the capacity to ground that text within visual context and reason coherently across both textual and visual evidence.

Why this matters#

VMMU addresses a critical gap in AI evaluation: most multimodal benchmarks focus on English-language content, leaving the real-world capabilities of VLMs in non-English settings largely unmeasured. By focusing on Vietnamese, the benchmark helps illuminate whether recent advances in vision-language understanding genuinely transfer to broader language diversity, or whether models have learned narrow patterns specific to English-centric training data. The code and dataset are publicly available for further research.