New metrics go beyond task success to measure robotic confidence and execution quality

Current approaches to testing vision-language-action (VLA) robots rely almost entirely on binary task success—did the robot complete its assigned job or not? A new study suggests this oversimplifies what actually matters: whether a robot executed a task well and how confident it was in doing so.

The limitations of binary evaluation#

Robots equipped with VLA capabilities can perceive their surroundings visually, understand natural language instructions, and plan actions to accomplish embodied tasks. Yet testing frameworks have remained primitive, typically recording only whether the robot succeeded or failed. This leaves crucial information hidden—a task might be technically completed sloppily, or a robot might succeed while remaining uncertain about its decisions. Neither dimension shows up in pass-fail metrics.

Testing new measurement approaches#

Researchers adapted eight uncertainty metrics and five quality metrics tailored to robotic manipulation work. They tested these across 908 successful task executions drawn from three leading VLA models, spanning four representative manipulation scenarios and two different robot designs. Human domain experts independently rated task quality on each trial, providing a gold standard against which to validate the new metrics.

Key findings#

The analysis revealed that multiple metrics demonstrated moderate to strong correlation with human expert judgment. This suggests they genuinely capture meaningful aspects of robot performance—not just noise. Additionally, the metrics showed promise at distinguishing high-, medium-, and low-quality executions even among tasks that the robot technically completed, suggesting utility for scenarios where human feedback isn't available.

Implications for robot development#

The findings indicate that relying solely on binary success rates leaves roboticists blind to important performance variation. By incorporating confidence and quality signals, developers could enable better real-time monitoring of robotic systems and support adaptive improvements to VLA models. The work opens a path toward more sophisticated evaluation frameworks that reflect what human users actually care about—not just whether a task got done, but how well it got done and whether the robot knew what it was doing.