New Benchmark Exposes How AI Vision Models Struggle With Geometric Puzzle Assembly

Recent research from arXiv has unveiled a notable gap in how advanced multimodal large language models (MLLMs) handle spatial reasoning. While these systems excel at visual recognition and semantic tasks, they falter when asked to solve geometry-based puzzles that require precise compositional reasoning.

The Problem with Existing Tests#

Current evaluation benchmarks for vision and language models tend to focus on coarse spatial relationships—asking whether one object is "to the left of" another, for instance. These assessments rarely enforce strict geometric verification or accommodate legitimate multiple solutions to construction problems, leaving a blind spot in how well models actually understand and work within geometric constraints.

Introducing TangramPuzzle#

To bridge this gap, researchers have developed TangramPuzzle, a new benchmark dataset containing 668 validated puzzle configurations and 1,336 test instances. The benchmark is specifically designed to probe how well MLLMs can reason about compositional spatial relationships when exact geometric rules apply.

The framework introduces the Tangram Construction Expression (TCE), a formal notation that encodes tangram puzzle configurations in ways that machines can verify automatically. Alongside this, the researchers built a multi-solution-aware verifier that independently checks two criteria:

• Geometric validity: whether piece arrangements respect the original shape constraints
• Silhouette fidelity: whether the assembled result matches the intended target outline

Two Complementary Tasks#

The benchmark evaluates models on complementary challenges:

• Outline Prediction: given puzzle pieces, infer the overall shape they should form (part-to-whole)
• End-to-End Assembly Generation: given a target silhouette, determine how pieces should be arranged to achieve it (whole-to-part)

What the Evaluation Reveals#

Testing across both open-source and proprietary state-of-the-art models uncovered a striking pattern: MLLMs tend to prioritize visually matching the target silhouette while sidelining geometric constraints. This trade-off manifests as piece distortions or deformations—solutions that correct but violate the underlying geometric rules.