New Method Speeds Up AI Interpretability Analysis with Near-Instant Attribution

Understanding what happens inside large language models remains a critical challenge for ensuring their safe and reliable use. A new research paper from arXiv proposes a solution that dramatically speeds up the process of attributing model outputs to specific internal components.

The Problem with Current Methods#

Existing attribution techniques struggle with a fundamental tradeoff: they can either explain what a model does accurately, or they can do so quickly—but rarely both. Dense component attribution, which aims to trace contributions across all layers and components, becomes prohibitively slow as models grow. This computational bottleneck makes it impractical to interpret modern transformers at scale.

How Dual Path Attribution Works#

The new approach, called Dual Path Attribution (DPA), reformulates the attribution problem by analytically decomposing SwiGLU Transformers into distinct information pathways. Rather than running multiple inference passes with counterfactual examples (the traditional approach), DPA propagates a target unembedding vector backward through the model in a single forward and backward pass. This target-centric strategy effectively reveals the contribution of each residual position to the final output.

The key insight is that DPA linearizes and simplifies the model's computational structure, allowing it to achieve O(1) time complexity relative to the number of model components—a dramatic theoretical improvement. This means attribution time barely increases as you probe deeper into the model or across longer inputs.

Empirical Results#

Testing on standard interpretability benchmarks, DPA demonstrates both superior accuracy (faithfulness) compared to existing methods and unprecedented computational efficiency. The technique scales naturally to long sequences and supports dense attribution analysis that previously would have been too expensive to compute.