New method improves diffusion language models by remasking low-confidence tokens instead of replacing them

Diffusion language models generate text through a denoising process that fills multiple masked positions simultaneously, but this approach can leave inconsistencies in place that affect downstream predictions. Researchers have developed a new method to address this fundamental problem.

The Core Challenge#

Diffusion language models differ from traditional autoregressive systems by predicting many tokens at once within each denoising step. Because all positions in a given step are filled without knowledge of each other's newly generated values, some predictions may conflict. Once these inconsistencies are locked in, they become the context for all subsequent predictions, potentially degrading output quality.

The Token-to-Mask Solution#

The new approach, called Token-to-Mask (T2M), operates at inference time without requiring model retraining. The method works by monitoring the model's confidence in each token through its probability scores, identifying positions where certainty is low. Rather than replacing uncertain tokens with alternatives, T2M remasks those positions so they can be reconstructed in later denoising steps with better contextual information.

Key Findings#

Testing across multiple diffusion language models showed several advantages:

• Transfer without tuning: A single T2M configuration works across different tasks and models without adjustment
• Broader improvements: The method consistently outperforms each model's built-in correction mechanism
• Efficiency over rewriting: When holding the detection method fixed, remasking yields better results than simply replacing suspicious tokens
• Competitive with autoregressive decoding: On most tested tasks, the approach retains sequential-step advantages while converting additional compute into measurable performance gains

The research decomposed correction into two components—detecting problematic tokens and deciding how to fix them—demonstrating that current-token-probability detection paired with remasking represents the most effective combination.