New Benchmark Reveals When Vision-Language Models Should Plan vs. Ground Symbolic Reasoning

Researchers have unveiled ViPlan, a new open-source benchmark designed to systematically evaluate how vision-language models (VLMs) can be paired with symbolic planning systems. The work addresses a gap in the research landscape: while integrating large language models with logical planners has shown promise, comparing these hybrid approaches fairly—and especially their visual variants—has lacked a standardized testing ground.

Two Competing Strategies#

The benchmark compares two distinct architectural approaches: using VLMs as grounders (extracting symbolic facts from images to feed into a classical planner) versus using VLMs as planners (generating action sequences directly from visual observations). Each strategy has different strengths and vulnerabilities.

Blocksworld Favors Symbolic Grounding#

In ViPlan's visual Blocksworld variant—a challenge domain involving the manipulation of colored blocks—the grounding approach significantly outperformed direct planning. VLM-as-grounder methods solved 46% of tasks, while VLM-as-planner approaches succeeded in just 9%. This gap emerged because the task's visual ground truth is precise and mostly observable; extracting clean symbolic facts from images and then applying rigid planning logic proved effective.

Household Robotics Shows the Opposite Pattern#

When tested on simulated household robotics tasks, the advantage flipped dramatically. Direct VLM planning succeeded 34% of the time, compared to only 5% for the grounding strategy. The researchers attribute this reversal to two factors: the richer linguistic knowledge embedded in VLMs becomes more valuable for reasoning about kitchen and household scenarios, and partial observability—where a robot cannot see all relevant aspects of its environment—undermines the grounding approach's reliance on visual extraction.

Broader Insights#

Across both domains, chain-of-thought prompting (asking models to reason step-by-step) offered no consistent benefit, pointing to deeper limitations in how current VLMs process visual information and plan actions. The benchmark thus captures fundamental trade-offs: grounding shines when images are clear and complete, but breaks down under partial observability and complexity, while direct planning leverages linguistic understanding but lacks verifiable grounding.