Large Language Models Fall Short in Robotic Planning When Paired With Formal Reasoning

Researchers have conducted an extensive evaluation of how large language models perform when applied to robotic planning tasks—finding that despite their semantic understanding, LLMs consistently underperform compared to hand-engineered planning systems.

The Study Design#

Scientists created 16 different algorithms that leverage LLMs to handle key components within task and motion planning (TAMP) frameworks. Across more than 13,750 zero-shot experiments spanning three distinct domains, they benchmarked these LLM-based approaches against established baselines.

Key Findings#

The results show clear gaps in LLM-based planning:

• Success rates: LLM planners achieved lower task completion rates than conventional engineered systems
• Planning speed: The systems required significantly more time to generate solutions
• Geometric complexity: When geometric details were included alongside planning descriptions, task-planning errors actually increased rather than improved
• Simpler beats complex: Direct LLM approaches—though faster—outperformed reasoning-heavy variants in most scenarios

The Integration Challenge#

Combining the semantic knowledge embedded in LLMs with the formal reasoning capabilities of structured planning systems remains conceptually attractive for robotics. Yet this investigation demonstrates that the execution involves numerous difficult design decisions about how to actually connect these two pieces. The variety of architectural choices leads to measurably different outcomes, with no single approach emerging as clearly superior across all conditions.

The research team has made both their code and experimental results publicly available, providing a foundation for future work on this integration challenge.