Classical animal learning theory explains how linear transformers attend to information

Researchers have uncovered a surprising bridge between modern machine learning and classical animal learning theory. The state updates that govern linear-attention transformer families follow the same mathematical patterns as century-old models used to describe how animals form learned associations—a connection that yields testable predictions about how these neural networks behave.

Mapping attention to conditioning#

The work draws a direct correspondence between three major attention mechanisms and foundational learning models. Linear attention implements Hebbian contiguity, the principle that co-occurring events become associated. DeltaNet mirrors the Rescorla–Wagner error-correction rule, which adjusts learning based on prediction mismatch. RetNet and related decay variants embody contiguity with a stimulus trace—the idea that learning involves temporal memory of past stimuli.

This dictionary translates well-established conditioning phenomena into concrete, testable predictions about how transformers process information in context.

What the theory predicts#

The researchers verified several predictions through simulation and training experiments:

• Kamin blocking: An algebraic closed form was derived for this classic conditioning effect and verified to precision better than 10⁻⁷ across multiple learning rates.
• Cue competition: Error-correcting attention (DeltaNet-like) exhibits cue competition—where one learned association interferes with another—while contiguity-based attention does not. This dissociation persists after training on generic in-context association tasks.
• Capacity regimes: A single state exhibits two distinct scaling behaviors: a measured exponent of 1.22 for accurate retrieval and 1.89 for identification, consistent with linear and near-quadratic theoretical predictions.
• Head architecture: Across the full attention head grid, retrieval error depends primarily on total state size rather than how that capacity is divided among heads, suggesting heads provide additional capacity rather than redundancy.

Theoretical and empirical findings#

A key algebraic result concerns spontaneous recovery—the reemergence of learned associations after extinction. The analysis proves no spontaneous recovery should occur for the single-state recurrences under examination when using orthogonal cue retention trials. Empirical tests with trained models, using never-before-presented-cue controls and probes within the trained range, confirmed this prediction.