Researchers Prove How Transformers Learn in Two Distinct Stages, Grounded in Feature Structure

Transformer models appear to follow a predictable two-phase learning pattern during training, yet this phenomenon has lacked rigorous mathematical explanation—until now. A new analysis from researchers on arXiv reveals how such staged learning emerges naturally when models must master multiple interdependent feature types.

The Two-Stage Pattern#

When training GPT-2 on factual knowledge tasks, an intriguing progression unfolds: answers first become syntactically sound before achieving semantic correctness. In other words, the model learns to produce grammatically valid responses before it learns to make those responses factually accurate. Similar staged behaviors appear across different domains—protein folding models learn primary structure before secondary structure, and natural language systems master form before meaning.

Despite widespread observation of this pattern in practice, existing theoretical frameworks have provided limited insight into why it occurs or how it relates to the underlying architecture of transformers.

A New Theoretical Foundation#

The research provides what is believed to be the first rigorous mathematical account of feature-level two-stage optimization in transformers. The analysis operates within a controlled setting: a simplified architecture using normalized ReLU self-attention and carefully structured input data that reflects real-world feature layering. By modeling features as fundamentally disentangled—such as syntax versus semantics—the researchers demonstrate mathematically how this structure inevitably drives staged learning dynamics.

The theoretical framework shows that such disentanglement is broadly applicable: natural languages inherently bundle syntax and semantics, biological sequences layer primary and secondary structures, and similar patterns likely exist across many domains where transformers are deployed.

Spectral Connections#

A key finding links the two-stage learning process to the spectral properties of attention weights—the eigenvalue characteristics that govern how attention mechanisms distribute information across the network. This connection suggests that understanding the mathematical properties of attention is essential to predicting and potentially controlling how models learn.

The work opens pathways for deeper understanding of transformer learning dynamics and may eventually enable better training procedures tailored to models with hierarchically organized feature spaces.