AI
Vision-Language Models Stumble on Visual Math Problems, Revealing Blind Spot in Reasoning
Researchers introduce a new benchmark that transforms text-based math problems into visual scenes, exposing a stark gap between how well AI models handle explicit symbols versus visual inference.
Vision-language models (VLMs) have shown impressive capabilities in many domains, but their ability to solve mathematical reasoning tasks that rely on visual perception remains unclear. A new benchmark study reveals a significant weakness: when math information must be inferred from images rather than read directly from text or symbols, top-performing models fail dramatically.
Introducing GSM8K-V#
Researchers have created GSM8K-V, a new benchmark dataset that adapts the popular GSM8K math problem suite into visual sequences. Rather than presenting word problems as plain text, the team mapped each problem into a series of images where quantities and relationships must be visually perceived and semantically interpreted. The process involved converting 1,319 problems through an automated pipeline followed by human review to ensure accuracy.
The key insight: answers cannot be simply read from explicit labels or symbols—instead, models must extract numbers through visual observation and piece together reasoning chains by connecting implicit cues scattered across multiple scenes.
The Modality Gap#
The results are striking. When tested on the original text-based GSM8K, most modern VLMs exceed 90% accuracy. However, when those same models face GSM8K-V's visual version, performance collapses. The highest-performing model achieved only 59% accuracy, far short of the 91% accuracy humans reach. This substantial gap persists even for models specifically enhanced for visual mathematical reasoning—suggesting this benchmark uncovers a distinct, previously unmeasured weakness.
The Core Problem: Implicit Visual Inference#
Error analysis points to a fundamental bottleneck: Implicit Visual Inference Error (IVIE). Models struggle when visual information is suggested rather than directly stated. They fail to recover the semantic meaning embedded in scenes, particularly when reasoning requires inferring quantities or relationships from context clues rather than explicit annotations.
This finding indicates that improving performance on existing visual math benchmarks has not translated into better real-world visual reasoning capabilities—the two appear to measure different skills entirely.
