New EM-based framework simplifies reinforcement learning for reasoning AI models

A new approach to training large language models for complex reasoning tasks could streamline current methods used in systems like OpenAI's O1 and DeepSeek-R1. Rather than relying on the heuristic adjustments found in existing algorithms, researchers have grounded their framework in classical statistical theory.

The Problem with Current Methods#

Widely adopted reinforcement learning algorithms such as PPO (Proximal Policy Optimization) and GRPO depend on two mechanisms that add unnecessary complexity: likelihood-ratio corrections and trajectory-level clipping. These tricks improve stability but introduce variance and potential optimization bias, making the algorithms harder to reason about and implement reliably.

A Principled Alternative#

The team introduces EM Policy Gradient (EMPG), which reformulates LLM reasoning training as an Expectation-Maximization problem. The approach unfolds in two steps: the E-step builds a reward-weighted distribution across reasoning paths (rationales), while the M-step refines the language model by optimizing the weighted likelihood of those paths. Because this formulation emerges directly from EM theory, it eliminates the need for trajectory importance weighting or clipping tricks.

Crucially, under exact computation of both steps, EMPG guarantees monotonic improvement in expected reward—a formal guarantee absent from heuristic methods.

Empirical Results#

Testing on GSM8K and MATH Hard benchmarks with Qwen2.5 models, EMPG matches or exceeds GRPO's performance while using a substantially cleaner optimization procedure. Beyond raw scores, the framework produced reasoning traces that were more concise and exhibited structured problem-solving behaviors: decomposing problems into subgoals, self-checking answers, reflecting on errors, and backtracking when needed.

Why It Matters#

This work demonstrates that viewing reasoning-focused reinforcement learning through the lens of classical statistics yields both theoretical clarity and practical gains. The EM perspective provides a foundation that is easier to analyze, modify, and build upon—offering a potential path forward as reasoning models grow more capable.