New policy gradient method trains diffusion language models more effectively on reasoning tasks

Diffusion-based language models—a class that generates text by progressively refining noisy samples—have emerged as a compelling counterpart to the dominant autoregressive architectures. Yet training them with reinforcement learning remains underdeveloped, holding back their performance on specialized reasoning problems.

A step-wise approach to policy optimization#

Researchers have introduced Amortized Group Relative Policy Optimization (AGRPO), a reinforcement learning method designed specifically for diffusion language models. Rather than treating each complete sequence as a monolithic unit, AGRPO exploits the iterative denoising nature of these models by optimizing one step at a time. This alignment between the training procedure and how the model actually generates text during inference eliminates a common source of bias.

The approach also introduces a novel timestep estimation technique that permits unbiased gradient updates—a technical refinement that makes the optimization process more stable and principled.

Strong empirical results#

On standardized benchmarks for structured reasoning, AGRPO demonstrated substantial improvements. Tested on the Countdown game and Sudoku puzzles using the LLaDA base model, the method achieved absolute accuracy gains of +59.4% and +69.7% respectively, surpassing related methods such as diffu-GRPO.

Why it matters#

The work addresses a gap in the diffusion language model ecosystem. While these models offer architectural advantages—such as parallel decoding potential and different inductive biases—they have lagged behind autoregressive competitors in real-world applications due to weak post-training techniques. By introducing an RL method that respects the Markovian structure of diffusion sampling, this research helps narrow that gap and unlocks reasoning capabilities in an underexplored model class.