New Training Approach Boosts Speed of AI Draft Models by 23% Over Competing Methods

A team of computer scientists has developed a new training strategy called Draft-OPD that significantly improves the performance of lightweight "draft" models used to speed up large language model inference. The approach achieves more than 5× lossless acceleration for reasoning models and outperforms existing techniques like EAGLE-3 and DFlash by 23% and 13% respectively.

How Speculative Decoding Works#

Speculative decoding is a technique that accelerates AI model inference by pairing a large, powerful target model with a smaller draft model. The draft model proposes tokens quickly, which are then verified in parallel by the target model, enabling faster token generation overall. This approach has become increasingly important as language models grow larger and slower.

The Challenge with Current Training Methods#

Most existing draft models are trained using supervised fine-tuning (SFT), where they learn from sequences generated by the target model. However, researchers identified a critical problem: SFT performance plateaus quickly. The draft model stops improving because of a mismatch between training and real-world use. During training, the drafter sees fixed, target-generated sequences. But during actual speculative decoding, it proposes tokens based on its own outputs—an "offline-to-inference mismatch" that causes it to encounter unfamiliar states it was never trained on.

The Draft-OPD Solution#

The new method, on-policy distillation (OPD), addresses this gap by having the target model supervise the drafter specifically on the states the draft model creates. However, naive on-policy learning failed because draft models cannot reliably complete sequences on their own. If they tried, the training data would drift away from the target distribution and eliminate the learning signal.

Draft-OPD solves this by combining two techniques:

• Target-assisted rollout: Using the target model to help complete sequences stably
• Error-position replay: Replaying drafting specifically from positions where verification revealed errors

This allows the drafter to learn from the target's feedback on both accepted and rejected proposals, focusing training effort on the exact draft-induced mistakes that limit speculative acceptance during real use.