New Diffusion Language Models Bridge the Gap Between Speed and Quality with Causal Attention

A new family of language models called Eso-LMs attempts to solve a persistent challenge in generative AI: delivering both fast, parallel generation and the perplexity performance of traditional sequential models.

The Problem with Current Alternatives#

Diffusion-based language models promise significant advantages over autoregressive (AR) models—they can generate text in parallel rather than token-by-token, and they offer better control over the output. However, existing masked diffusion models (MDMs) still lag behind AR systems in terms of perplexity, a key measure of language model quality. More critically, they lack KV caching, an optimization that dramatically speeds up inference by storing previously computed values.

This limitation creates a fundamental trade-off: choose speed and controllability, or choose quality and efficiency.

How Eso-LMs Work#

Rather than relying on bidirectional attention like prior MDM approaches, the Eso-LM architecture leverages a connection between masked diffusion and any-order autoregressive models, adopting causal attention instead. This shift is crucial—it allows the framework to compute exact likelihood values for diffusion models for the first time.

More importantly, introducing causal attention enables KV caching for diffusion models while maintaining parallel generation capabilities. This dual achievement represents a significant technical breakthrough: inference becomes far more efficient without sacrificing the speed advantages that make diffusion models attractive.

Key Advances#

The research introduces several improvements:

• Exact likelihood computation for masked diffusion models
• KV caching support for diffusion-based generation, improving inference speed
• Parallel generation capability preserved despite efficiency gains
• An optimized sampling schedule that further improves performance

Together, these changes establish a new benchmark on the speed-quality trade-off frontier for unconditional text generation.

What's Next#

The authors have made code, trained model checkpoints, and a video tutorial available through their project page, enabling other researchers to build on this work.