New AI Framework Automates Test Oracle Creation for Robot Vision-Language Models

Researchers have developed MANGO, an automated framework that addresses a longstanding challenge in robotics testing: creating test oracles—decision-making standards that determine whether a robotic task has succeeded. Rather than relying on hand-built symbolic rules that are expensive, expertise-heavy, and task-specific, MANGO generates executable oracles automatically from plain-language task descriptions.

The Problem with Manual Oracles#

Vision-Language-Action (VLA) models are integrated systems that allow robots to perceive their environment, understand instructions, and execute physical actions. However, testing these systems has relied on manually crafted symbolic oracles tied to particular tasks and environments. This approach creates several bottlenecks: domain specialists must write them, they don't transfer across different scenarios, and they only check whether a task succeeded or failed at the end—without revealing where things went wrong or how the robot behaved along the way.

How MANGO Works#

The framework employs three collaborative AI agents—a Generator, an Assessor, and a Judge—that work iteratively to build a robust testing infrastructure. First, MANGO constructs a reusable library of simple, atomic tasks by breaking down complex robotic operations into their building blocks. Next, it generates oracle definitions for each atomic task, grounding them in a simulator so they can be validated. Finally, it decomposes complex instructions into ordered sequences of atomic actions, each with a corresponding oracle, creating fine-grained test coverage that can pinpoint exactly where a task fails.

Evaluation Results#

Testing on standard benchmarks—LIBERO_10 and RoboCasa Humanoid Tabletop—demonstrated that MANGO's automatically generated oracles detect as many failures as traditional symbolic oracles while providing better fault localization and deeper diagnostic insights. Ablation studies confirmed that the framework's core components each contribute meaningfully to oracle quality, and the system performs well even with different initial task sets.

Implications for Robotic Testing#

By automating oracle generation and making test artifacts reusable across tasks and environments, MANGO lowers the barrier to comprehensive robot testing. The approach shows that multi-agent AI collaboration can solve a real engineering bottleneck in the deployment and validation of complex robotic systems.