New technique helps diffusion language models improve themselves without extra training

Diffusion-based large language models (dLLMs) have proven capable at complex reasoning tasks, but enhancing their performance typically demands expensive post-training with new datasets and supervision. A research team now demonstrates that such models can enhance themselves during inference—without additional training, data collection, or reward models.

The core innovation#

The key challenge lies in generating a reliable guidance signal for partially masked intermediate states in dLLMs, something previous approaches have struggled to do. Researchers propose Reward-Free Guidance (RFG), a training-free framework that extracts improvement signals directly from existing model checkpoints.

The method is grounded in theory: a reward signal for partially masked states can be expressed mathematically as the log-likelihood ratio between a fine-tuned policy dLLM and its base reference model. Crucially, practitioners can calculate this signal using standard checkpoints alone, making the approach highly practical.

Performance gains#

Experimental results demonstrate that RFG consistently boosts state-of-the-art post-trained dLLMs by up to 16.1%. Remarkably, despite requiring no training resources, the improvements match or exceed those achieved through computationally intensive reinforcement learning methods.

Why it matters#

This approach reduces barriers to model improvement by eliminating the need for costly post-training infrastructure, additional labeled data, or separately trained reward models. By working exclusively with checkpoints already in hand, RFG opens a path for practitioners to strengthen their models more efficiently.