New Machine Learning Method Generates Diverse Robot Behaviors From Demonstration Data

Researchers have developed a new approach to extracting diverse behavioral strategies from recorded demonstration data without requiring further interaction with the environment. The technique, called Dual-Force, combines a physics-inspired objective with a novel encoding method to overcome practical challenges that plague existing offline learning systems.

The Problem with Current Methods#

Existing offline diversity maximization techniques typically rely on mutual-information objectives that require training an auxiliary component called a skill discriminator. This architecture introduces computational overhead and stability issues. When the system uses alternating optimization to enforce behavioral constraints, the changing reward signals cause training to become unpredictable and difficult to manage.

How Dual-Force Works#

The approach introduces two key innovations:

• Physics-Inspired Diversity: Rather than using traditional mutual-information measures, Dual-Force draws on a Van der Waals force objective—a concept from physics—computed from successor features. This eliminates the need for a skill discriminator altogether, simplifying the overall architecture.

• Functional Reward Encoding: A pre-trained component called Functional Reward Encoding (FRE) stabilizes training by conditioning both the value function and policy on consistent representations. The FRE also stores learned skills as latent codes, enabling instant recall of any previously encountered behavior without needing to specify the number of skills in advance.

Real-World Testing#

The researchers evaluated Dual-Force on Solo12 robot simulations in two tasks: locomotion and obstacle navigation. The algorithm successfully learned multiple high-performing behaviors while matching expert-level state occupancy patterns, and it demonstrated improved resilience when obstacles changed unexpectedly during testing.

View the full paper on arXiv