Researchers develop tractable reinforcement learning framework under unpredictable external shocks

Researchers have tackled a fundamental challenge in reinforcement learning: making optimal decisions when outside forces randomly disrupt an agent's environment in unpredictable ways. The work, submitted to arXiv, addresses how machines can learn effective policies when facing perturbations that don't follow standard Markovian patterns.

The Problem#

Traditional reinforcement learning assumes that future outcomes depend only on the current state—a property known as the Markov assumption. Real-world systems rarely work that way. External events like market shocks, weather disruptions, or network failures can create cascading effects with long-range dependencies. The paper explores how to handle these non-Markovian disturbances in settings with continuous state and action spaces.

Key Findings#

The researchers established conditions under which the problem becomes mathematically tractable by demonstrating that agents only need to track a finite history of prior external events rather than an infinite record. This simplification is grounded in properties of how external perturbations decay over time.

The team developed a policy iteration algorithm that learns decision rules based both on the environment's current state and a manageable window of past exogenous shocks. Because convergence isn't guaranteed, they provide instead a bound on policy improvement within regions where their approximations work best. They also quantified the sample complexity—how much data is needed—for both policy evaluation and improvement under these constraints.

Scope and Application#

While the results apply broadly to discrete-time processes meeting certain decay conditions, the authors focused on a specific case: Hawkes processes with Gaussian marks, a well-studied model for capturing self-exciting event sequences. Their experiments validated the approach in standard control environments, demonstrating that policies trained under this framework perform effectively when deployed.