New Framework Splits Robotic Vision and Planning to Boost Manipulation Tasks

A team of researchers has unveiled Gondola, a modular system that tackles long-horizon robotic manipulation by decoupling visual understanding from action execution. Rather than forcing a single model to directly convert images and text commands into low-level motor controls—an approach that often lacks transparency and fails on complex tasks—the framework first generates a structured, human-readable plan with explicit reference to specific objects and target locations before any action occurs.

How Gondola Works#

The system accepts multi-view camera feeds and planning history, then outputs the next step as a combination of textual instructions and segmentation masks that pinpoint which objects to manipulate and where to move them. This intermediate representation creates a clear bridge between what the robot "understands" about the scene and what it actually does, making the decision-making process interpretable and easier to debug.

Training leverages synthetic datasets that provide supervised examples across three key capabilities: short-horizon grounded planning (identifying objects in the immediate next step), multi-view language understanding (recognizing objects from different angles), and long-horizon compositional reasoning (breaking multi-step tasks into coherent sub-goals). This targeted supervision ensures the planner learns to communicate unambiguously about spatial intent.

Performance and Real-World Transfer#

When paired with a 3D-aware execution policy, the framework achieves top results on GemBench, a benchmark designed to stress-test robotic systems on demanding manipulation scenarios. Testing also shows promising results when deployed on physical robots, suggesting the approach generalizes beyond the lab setting. Ablation studies underline that both the pixel-level grounding and the planning-oriented training strategy are essential—dropping either one significantly degrades performance.

By making the reasoning process explicit and modular, Gondola addresses a fundamental challenge in robotic learning: the need for systems that are not just accurate, but interpretable and robust enough for real-world deployment.