New Distillation Technique Speeds Up Language Models Without Sacrificing Quality

Traditional autoregressive language models generate text one token at a time, which is accurate but slow. Researchers have developed parallel decoding methods that produce multiple tokens simultaneously, but these faster approaches tend to produce lower-quality outputs. A new technique called Gumbel Distillation aims to bridge that gap.

How It Works#

The method uses a mathematical approach called the Gumbel-Max trick to create a deterministic mapping between random noise and output tokens from a high-performing teacher model. This allows parallel decoders to learn the complex patterns in how words and tokens naturally relate to one another in sequences. Because it doesn't depend on any specific model architecture, the technique can work with various parallel decoding systems like MDLM and BD3-LM.

Demonstrated Improvements#

Testing on standard benchmarks—LM1B and OpenWebText datasets—showed substantial gains. On the OpenWebText dataset, the approach achieved a 30.0% improvement in MAUVE score (a measure of text quality similarity) and a 10.5% improvement in generative perplexity compared to MDLM baseline models. These results suggest the technique successfully narrows the quality gap between fast parallel generation and slower but more accurate sequential models.

Practical Impact#

The model-agnostic design means Gumbel Distillation can be incorporated into different parallel decoding frameworks without major architectural changes, making it potentially useful across a range of language model applications where speed and quality both matter.