Startup Builds Robots That Learn New Tasks From a Single Video

A Cambridge-based robotics startup called Generalist AI is developing machine arms that absorb new instructions remarkably fast, absorbing how to tackle unfamiliar jobs after watching a single instructional clip and without any specialized preparation for each individual task.

How the robots learn#

The approach draws on how humans and young children grasp physical concepts intuitively from their surroundings. Rather than requiring thousands of labeled examples to master a single activity—the traditional method in robotics—these systems appear to transfer knowledge between scenarios. Generalist's team, which includes founding members from Google DeepMind and Boston Dynamics, built their own AI models from the ground up instead of repurposing existing open-source language models.

Demonstrations show real-world flexibility#

The systems have shown surprising ingenuity in real demonstrations. When a robot tasked with moving a block into a bowl lost access to a brush, it grabbed a dustpan and used that instead. In another example, a two-armed system watched a human unzip a wallet and extract cash—then successfully unzipped a different purse and retrieved bills from it. When standard gripping didn't work, the robot switched from its right gripper to its left to reach from a better angle.

CEO Pete Florence drew a parallel to GPT-3's capabilities: "You could take that model and just prompt it to do a new task and it would have a real shot at doing it." The robots occasionally surprise their engineers—one spontaneously decided to sweep objects with a banana when it spotted one.

Data-driven approach#

Generalist collects training data using custom gloves fitted with cameras that mirror robot grippers. Human operators wear these to demonstrate tasks, with hundreds of such devices already deployed to workers in Mexico and elsewhere. The company has amassed substantial high-quality training data this way, though details about the exact learning methods remain undisclosed.

Remaining challenges#

The technology is not yet production-ready. A robot successfully completes tasks it has been shown only around 59 percent of the time on average—well below the 99 percent reliability manufacturers would likely demand. Questions also remain about whether these learned behaviors will scale across vastly different environments and task types.