New benchmark reveals gaps in AI vision-language models for assembly task understanding

Researchers have developed a new benchmark called LEGO Co-builder to evaluate how well current vision-language models (VLMs) can understand and follow step-by-step assembly instructions—a task that demands precise spatial reasoning and careful tracking of how objects change state.

The challenge#

Even sophisticated AI models struggle when asked to understand fine-grained visual details and spatial relationships in complex assembly scenarios. While systems like GPT-4o and Gemini excel at many visual reasoning tasks, they falter when confronted with detailed procedural instructions that require both seeing what's happening and predicting what comes next in a physical assembly workflow.

How the benchmark works#

The LEGO Co-builder benchmark combines real-world assembly logic with computer-generated multimodal scenes. It captures each step of an assembly process alongside both visual states and procedural instructions, creating a controlled testing environment. This hybrid approach allows researchers to systematically evaluate three key capabilities: whether models can follow instructions correctly, detect specific objects in scenes, and recognize state changes as assembly progresses.

What the tests revealed#

The team assessed several leading models—including GPT-4o, Gemini, Qwen-VL, and the reasoning-focused GLM-4.1-thinking—both without fine-tuning and with specialized training. Results showed a stark performance gap:

• Object detection performed well, with fine-tuned InstructBLIP reaching 98.16% accuracy
• Identifying theme entities proved much harder, with fine-tuned MiniGPT-v2 achieving only 37.52% F1 score
• State detection, where models must recognize how conditions have changed, saw GPT-4o reach just 40.54% F1

These numbers suggest that while models can identify individual components, they struggle to understand scenes holistically and track how assembly states evolve step by step.

What's next#

The researchers have released their benchmark, code, and generation pipeline publicly to accelerate development of better multimodal assembly assistants. The work points to a concrete research direction: building AI systems that can match human-level reasoning about spatial relationships and procedural sequences in practical, real-world contexts.