New VLM-Guided Framework Uses Rendered Robot Motions to Improve Real-World Manipulation Tasks

A team of roboticists has developed AntiGrounding, a framework that bridges the gap between language instructions and executable robot movements by leveraging vision-language models to assess candidate trajectories. Rather than asking a VLM to generate motions directly, the system renders short trajectory segments as visual inputs and uses the model's reasoning capabilities to score and rank them.

How It Works#

The approach operates on a two-stage principle: first, candidate trajectories pass through feasibility filters to remove physically implausible options. Each surviving trajectory then serves dual purposes—it remains a concrete motion plan ready for execution while also being visualized as a prompt for a vision-language model evaluator.

Structured multi-view visual question answering assesses trajectories across multiple dimensions: safety, task alignment, efficiency, and physical plausibility. The system treats different view angles as distinct evaluation perspectives and combines their scores through weighted fusion. This combined score guides the selection and refinement of subsequent trajectory proposals. Motion control is decomposed into separate handling for translation, orientation, and gripper actuation to improve modularity.

Validation Before Execution#

Before the physical robot attempts any motion, an initialized digital twin environment simulates and validates the chosen trajectory segments. This intermediate step helps catch potential failures in the controlled software environment, reducing costly real-world errors.

Results on Real Robots#

Testing across eight real-world manipulation tasks demonstrated substantial improvements in execution success. AntiGrounding achieved 71.25% overall success, outperforming baseline methods: pi0.5 reached 50.00% and a PIVOT-style visual selection approach achieved 47.50%, both operating under identical evaluation conditions. The framework showed robustness across ablations targeting trajectory evaluation, proposal search mechanisms, orientation control, and alternative evaluator models.

Current Limitations#

The method's performance ceiling appears bounded by the fidelity of the digital twin simulation and the physical interaction capabilities of the robotic platform. Despite these constraints, the framework demonstrates how general-purpose multimodal reasoning can be effectively grounded in executable robot actions.