New Speculative Decoding Method Uses Smart Vocabulary Selection to Speed Up AI Text Generation

Speculative decoding has become a popular technique for making large language models faster without changing their outputs. The approach uses a lightweight draft model to predict what a larger target model will produce, then verifies these predictions to save computation.

The Vocabulary Bottleneck#

Current best-in-class methods like EAGLE-3 rely on a minimal draft model containing just a single decoder layer. However, the output embedding matrix—which maps predictions to actual token distributions—becomes a computational bottleneck, especially as models grow larger. Some recent approaches have tackled this by shrinking the draft model's vocabulary, reducing the number of tokens it needs to consider during prediction. While this speeds up the drafting phase, there's a tradeoff: if the target model wants to produce a token outside this restricted vocabulary, the speculation fails and the system must fall back to slower methods.

SpecVocab: Dynamic Vocabulary Speculation#

Rather than permanently limiting the draft model's vocabulary, researchers propose an alternative: select a different vocabulary subset at each decoding step. The new method, called SpecVocab, intelligently chooses which tokens to consider during inference based on the current context. This adaptive approach maintains speculation effectiveness even when the final chosen token might have been excluded from a static reduced vocabulary.

Performance Gains#

Across a range of language modeling tasks, SpecVocab outperforms EAGLE-3 in acceptance length—the number of tokens successfully predicted and verified in a single speculation round. This improvement translates to tangible speed gains, with throughput increases of up to 8.1% compared to the prior state-of-the-art method. The technique balances computational efficiency with the ability to handle the full target model's output space.