AI
New Benchmark Exposes Major Gaps in AI Vision Models' Spatial Reasoning
Researchers testing multimodal AI systems on LEGO-based spatial puzzles find even top models fail at basic assembly tasks where humans excel.
Researchers have unveiled a new benchmark that reveals stark limitations in how well today's leading multimodal AI models can handle spatial reasoning—a fundamental skill needed for robotics, autonomous vehicles, and assembly automation.
The LEGO-Puzzles Benchmark#
The benchmark, called LEGO-Puzzles, uses LEGO construction as a testing ground for spatial intelligence. The system is organized into two task sets: an Elementary set with 1,100 visual question-answering samples that probe foundational spatial skills, and a Planning set that asks models to generate step-by-step assembly instructions for LEGO structures with planning horizons extending up to 8 steps.
The design is intentional—LEGO construction inherently demands that builders understand spatial relationships, remember previous assembly steps, and plan ahead for the next move. These are precisely the capabilities that autonomous systems need.
Where AI Models Fall Short#
Testing across 29 state-of-the-art multimodal large language models (MLLMs) painted a sobering picture. Even the strongest performers underperformed human participants by at least 20 percentage points on elementary spatial reasoning tasks. When researchers asked models to generate multi-step plans, accuracy plummeted rapidly—collapsing to zero as the planning horizon grew. In contrast, human participants solved all tasks without error.
The problem worsened when researchers changed the output format. Switching from multiple-choice answers to having models generate images directly produced even worse results, with zero accuracy achieved on planning tasks requiring just three steps ahead.
Why This Matters#
These findings underscore a critical gap between current AI capabilities and what's needed for real-world spatial tasks. Applications ranging from robotic assembly lines to autonomous vehicle navigation depend on the kind of multi-step spatial understanding that these tests measure. The benchmark reveals that today's advanced models lack the reasoning depth required for these safety-critical domains.
