New routing method uses LLM internal activations to predict which model handles tasks best

A new approach to routing queries across multiple language models leverages the models' internal computational patterns rather than explicit task features. Researchers demonstrate that signals drawn from a model's intermediate layer outputs—specifically the residual stream during the initial processing phase—can effectively predict which model will perform best on any given input.

The Core Innovation#

The key breakthrough is Encoder-Target Decoupling, a framework that separates the source of prediction signals from the target model being evaluated. This means an open-weight encoder can generate reliability predictions about the performance of closed-source models, breaking down barriers that typically prevent cross-model routing systems.

The team identifies that certain layers within language models are particularly informative for this task. Using Fisher Separability and Effective Dimensionality measurements, they pinpoint which layers contain the strongest predictive signals about model success or failure.

The Implementation#

The proposed SharedTrunkNet architecture uses a single multi-output neural network that processes concatenated activation features from the prefill phase—the initial pass when a model encodes the input prompt. Rather than making separate predictions for each candidate model, SharedTrunkNet predicts correctness probabilities across all available options simultaneously, creating a unified routing decision.

Performance Results#

In experiments, the activation-based approach significantly outperforms baselines relying on semantic similarity or handcrafted heuristics. Most notably, SharedTrunkNet closed nearly half the performance gap between the strongest single model and an oracle selection system while reducing computational costs by over 74% compared to running the most expensive model unconditionally.

This suggests that the patterns models learn internally—how they organize information during processing—encode richer signals about task difficulty and failure modes than explicit semantic features alone.