Mixture-of-Depths Ensemble: Tapping Late-Layer Intelligence in LLM Tuning

Researchers working on Transformer-based large language models have typically relied on tuning losses from the final layer and using final-layer representations for predictions. A new approach challenges this conventional wisdom by asking: what if the task-relevant signals already forming in earlier layers could be harnessed directly?

The Core Insight#

Interpretability tools like the logit lens have revealed that late-layer representations already contain largely-formed, task-relevant predictions. The key question became whether this observation could translate into a useful training signal. The research from arXiv demonstrates that focusing tuning effort on these intermediate layers can achieve losses comparable to final-layer tuning, with complementary behavior at test time.

Introducing Mixture-of-Depths Ensemble#

Building on this foundation, the team introduces Mixture-of-Depths Ensemble (MoDE), a tuning framework that treats late layers as an ensemble. Rather than discarding their outputs, MoDE routes learned weights through these layers to contribute directly to final logits. The approach is agnostic to the underlying tuning method—it can be applied on top of existing techniques like LoRA.

Results and Implications#

In experiments, MoDE modestly improves reasoning performance with only a small parameter overhead. More importantly, it establishes that late-layer logits can be made directly useful for tuning and can substitute for substantially larger trainable modules while maintaining comparable performance. This work functions as a mechanism study, revealing untapped capacity in intermediate network layers that could inform more efficient model adaptation strategies.