AI
New benchmark reveals gap between visual polish and reasoning in AI image and video generators
Researchers introduce MMGR to evaluate whether multimodal AI systems can solve reasoning tasks or merely produce plausible-looking content.
A team of researchers has released a new evaluation framework that challenges a common assumption about modern generative AI: that visually convincing outputs necessarily reflect sound reasoning ability. The MMGR (Multi-Modal Generative Reasoning Benchmark and Evaluation) directly tests whether state-of-the-art video, image, and language models can maintain physical, logical, spatial, and temporal constraints when solving problems—or if they simply generate content that looks plausible.
The Problem with Visual Fluency#
Today's generative systems produce remarkably realistic images and videos, but visual quality alone tells us little about whether a model actually understands the underlying problem. A system might generate a smooth, aesthetically coherent video sequence without those frames forming a logically valid solution. The new benchmark shifts evaluation focus from "does this look good?" to "does this actually solve the task?"
What MMGR Tests#
The benchmark spans 10 tasks across three domains:
- Abstract Reasoning — including pattern completion and logic puzzles
- Embodied Navigation — simulated spatial reasoning and movement
- Physical Commonsense — understanding how objects behave in the real world
It probes five distinct reasoning dimensions: physical constraints, logical consistency, 2D spatial awareness, 3D spatial understanding, and temporal coherence. For video generation specifically, the framework emphasizes "chain-of-frame reasoning"—intermediate frames must constitute valid steps toward the solution, not merely smooth visual transitions between unrelated states.
Key Findings#
Testing leading video generators, image generators, and LLM/VLM systems revealed striking disparities:
- Video models excel at physical commonsense tasks but falter on symbolic problems like Sudoku, ARC, and math reasoning
- Cross-view embodied navigation exposes brittleness in systems that seemed otherwise capable
- Longer video output does not guarantee stronger reasoning—some image generators outperformed video models on navigation tasks despite producing only static outputs
