New AI Framework Uses Self-Invented Language to Improve Reinforcement Learning Efficiency

Researchers have developed a novel approach to making reinforcement learning agents more sample-efficient when following natural language instructions. The challenge lies in adapting Hindsight Experience Replay (HER)—a technique that boosts learning by repurposing failed attempts with different goals—to work with language-based objectives.

The Core Problem#

Standard HER requires access to two key functions: one that can relabel failed trajectories with goals the agent actually achieved, and another that can verify whether a goal was satisfied. This works well in environments with explicit numeric goals, but breaks down when goals are expressed in natural language instructions that don't directly correspond to the agent's state space. Researchers framed this gap as the Hindsight Reinforcement Learning problem, highlighting the need to learn these functions jointly with the policy itself.

How ETHER Works#

The proposed system, called ETHER (Emergent Textual Hindsight Experience Replay), tackles this by combining two separate research areas: emergent communication and goal-conditioned reinforcement learning.

The approach trains agents to develop their own artificial language through a referential game—one agent (the speaker) learns to describe environment states in a made-up language, while another (the listener) learns to interpret those descriptions. This grounded communication system is then partially aligned with natural language instructions by leveraging patterns between task instructions and the agent's observations in the environment.

Results and Implications#

Testing on the BabyAI benchmark revealed that the speaker and listener components learned through the referential game can effectively stand in for HER's missing functions. Even with imperfect alignment between the emergent language and human instructions, the method improved sample efficiency in learning.

By connecting emergent communication research with practical reinforcement learning challenges, ETHER opens possibilities for applying HER—and its sample-efficiency gains—to a broader class of language-based tasks.