First watermarking technique for diffusion language models tackles out-of-order token generation

A new watermarking approach addresses a fundamental mismatch between emerging diffusion language models (DLMs) and existing watermarking techniques designed for conventional autoregressive models.

The challenge of out-of-order generation#

Diffusion language models represent a different computational paradigm than the sequential token generation used by standard LLMs. While autoregressive language models produce text one token at a time in a fixed order, DLMs can generate tokens in arbitrary sequences. This flexibility creates a problem: established watermarking methods assume that previous tokens are already known when deciding which new token to generate, making them incompatible with the DLM approach.

How the new watermark works#

The researchers address this constraint through two key innovations. First, they apply the watermark by calculating its expected value across possible contexts, even when some of those context tokens haven't yet been determined. Second, they strategically promote tokens that strengthen the watermark effect when those tokens later serve as context for other generation steps. Notably, the detection mechanism remains entirely unchanged, simplifying deployment.

Performance results#

Testing reveals strong performance metrics: the watermark achieves over 99% true positive detection rates while maintaining minimal degradation to output quality. The robustness properties match those of established watermarks for autoregressive models, making it the first practical watermarking solution tailored for the DLM paradigm.