TIDE: Self-Optimizing Inference Engine Speeds Up Large Language Models

Researchers have proposed TIDE (Temporal Incremental Draft Engine), a framework designed to speed up large language model inference by embedding adaptive draft model training directly into inference systems. The approach addresses a practical bottleneck: while speculative decoding techniques can theoretically accelerate LLM serving, they often falter when actual usage patterns diverge from the assumptions built into static draft models.

How TIDE Works#

TIDE integrates online learning into the inference pipeline by repurposing hidden states generated during normal inference as training signals for improving draft models. This eliminates the need for separate target model computations, keeping the system efficient. The framework includes adaptive runtime control that intelligently decides when to activate speculation and draft training—only engaging these features when they promise tangible benefits rather than wasting resources on every query.

To handle diverse hardware setups, TIDE maps inference and training tasks to the most appropriate GPU types within a cluster, taking advantage of heterogeneous infrastructure in real-world data centers.

Performance Gains#

Across real-world workloads, TIDE achieves up to 1.66× throughput improvement compared to systems without speculation. Crucially, the approach recovers performance in scenarios where traditional static draft models actually hurt throughput due to misalignment with actual traffic patterns.

The framework also delivers substantial engineering improvements: draft model training time drops by up to 3.02×, storage overhead shrinks by 24×, and system throughput on mixed-GPU clusters improves by up to 1.22× versus existing approaches.

Why It Matters#

Speculative decoding remains one of the most practical techniques for accelerating LLM inference in production. By making draft models self-adapting rather than fixed, TIDE bridges the gap between theoretical potential and real-world deployment, where workloads constantly shift.