New decoder technique speeds up LLM inference by extracting hidden predictions mid-computation

Researchers have developed a method to extract reliable predictions from large language models partway through their computation, potentially cutting inference time by up to 40 percent for certain tasks. The technique, called SimLens, reads the model's internal representations at early layers and uses a lightweight continuation step to decode what the model will eventually decide—without computing all the way to the final layer.

The problem with reading hidden layers#

Intermediate-layer representations in LLMs contain useful information, but they're notoriously difficult to interpret. Previous approaches relied on direct linear decoding—essentially treating internal states as raw features and projecting them directly to an answer. This straightforward strategy often drifts from the model's final prediction because early layers haven't yet consolidated all the reasoning needed for an accurate decision.

How SimLens works#

Rather than attempting direct decoding, SimLens takes a minimal but powerful approach: it keeps only two tokens—the starting token and a candidate answer—and runs them through the remaining upper layers of the model. This lightweight continuation captures the model's latent reasoning process with minimal extra computation. The technique requires no retraining and works immediately with existing models.

The researchers also developed Linear SimLens, a fast linear approximation for estimating confidence in the model's predictions based on entropy, and combined both into SimExit, a hybrid early-exit system that decides whether to stop computation early or proceed to deeper layers.

Performance gains across benchmarks#

Testing on reading comprehension (ARC, BoolQ) and medical QA (HeadQA) tasks with LLaMA-7B and Vicuna-7B, SimLens improved accuracy on all six configurations. Accounting for the overhead of the two-token continuation, the method achieved an average +0.43 accuracy gain compared to standard approaches. SimExit delivered 1.15× speedup at optimal accuracy and 1.40× speedup when tolerating a 1 percentage-point drop, enabling substantially faster inference on single-token decision tasks.

Why this matters#

Efficient inference is critical for deploying LLMs at scale. By reading early-layer signals accurately, this approach lets systems confidently exit computation before reaching the final layer, reducing computational cost without sacrificing answer quality—a practical step toward making large models more resource-efficient.