Video game data could train the next generation of AI robots and autonomous systems

Large language models excel at processing text, but they fall short when it comes to understanding the physical world. Training an AI to pilot a robot or guide an autonomous vehicle demands something fundamentally different: exposure to how actions—button presses, joystick movements, grip strength—produce real-world consequences.

Researchers including Fei-Fei Li and Yann LeCun are now turning to world models, a class of AI designed to learn physical cause-and-effect relationships by ingesting both visual data and action information. Yet there's a critical problem: unlike text, which fills the internet in staggering volumes, authentic training footage showing objects being manipulated with precise details isn't readily available.

Where video games fit in#

A startup called Worldmodeldata, advised by Yann LeCun, proposes an unconventional solution. The company is licensing massive libraries of video game data—controller inputs combined with visual output—from studios as training material for world models. Rather than striking deals with countless game makers individually, Worldmodeldata positions itself as a curator and broker for this largely untapped resource.

"There are millions of great games, and they are more and more similar to the real world," CEO Rhea Loucas explains. The startup has already licensed nearly 1 million hours of gameplay footage and is exploring ways to compensate individual players in the future.

Worldmodeldata's bet mirrors the scaling success of language models: better training datasets should yield better performance. The advantage of game data is its sheer volume and diversity—millions of scenarios, edge cases, and unusual situations that might be prohibitively expensive to capture in controlled laboratory settings.

Skeptics raise concerns#

Not everyone believes video games are the ideal training ground. Nvidia, which develops its own family of world models, uses a custom physics engine instead. Ming-Yu Liu, who leads Nvidia's world model team, warns that game physics often rely on shortcuts and approximations—a character's hand might appear to grip an apple without the model accounting for finger pressure or friction. For fine-grained manipulation tasks, these omissions could prove disqualifying.

Xiatian Zhu, an AI researcher at the University of Surrey, echoes the concern: "Video games are, in essence, simulators. They do have some degree of physical grounding, but they are very coarse, approximate." Real-world physics for delicate operations like robotic assembly are far more intricate than what games simulate.