New method teaches AI agents to learn varied, interpretable strategies through temporal abstraction

A core challenge in reinforcement learning is enabling agents to reason and act across varying time horizons—a capability called temporal abstraction. The option-critic framework has shown promise in this area by allowing agents to learn extended sequences of actions, termed options, in an end-to-end fashion without requiring explicit models of the environment.

The core problem#

However, the option-critic approach faces significant limitations. When trained, agents often converge on options that exhibit redundant or nearly identical behaviors, negating the intended benefit of temporal abstraction. Simultaneously, the set of learned options tends to shrink over training, leaving the agent with fewer distinct action strategies available. Both phenomena degrade performance and undermine the framework's theoretical promise.

Introducing diversity-enriched learning#

To address these issues, researchers have developed an augmented approach that actively promotes behavioral diversity among options. The method combines two key innovations: an intrinsic reward signal grounded in information theory, which supplements the standard task reward, and a redesigned termination objective that discourages options from behaving identically.

Empirical results#

Testing across discrete and continuous control environments, the enhanced approach substantially outperforms the baseline option-critic method. Crucially, the learned options exhibit properties absent in standard implementations—they prove robust to environmental variations, can be reused across related tasks, remain interpretable to human observers, and are demonstrably reliable in deployment. These characteristics make the options genuinely useful building blocks for hierarchical decision-making, rather than artifacts of an incomplete learning process.

The work, submitted to arXiv in November 2020 and revised in July 2026, advances the practical viability of temporal abstraction in reinforcement learning agents.