Researchers map how language models learn and forget concepts during continued training

A team of researchers has opened a window into how large language models acquire, retain, and lose conceptual knowledge as they continue to learn from new data. By examining the internal mechanisms that underpin concept formation, the work offers fresh insights into the hidden processes shaping LLM behavior.

How LLMs represent concepts internally#

The study identifies concept circuits—specific patterns of computational connections within a model that correspond to individual concepts like "dog" or "justice." Rather than treating LLMs as black boxes, the researchers traced these internal pathways and used graph-based metrics to describe their structure and topology. This circuit-level approach revealed that these internal representations consistently reflect whether and how well a concept is being learned or forgotten.

The lifecycle of concept learning#

As LLMs undergo continual pretraining on new data, their concept circuits follow a predictable temporal pattern. Concepts initially show rapid gains in activation and connectivity, then gradually decline and eventually stabilize. Notably, concepts that achieve the strongest learning gains tend to experience the most forgetting when exposed to subsequent training, suggesting a potential trade-off between initial acquisition and long-term retention.

How concepts interact and interfere#

The research illuminated how multiple concepts influence one another during learning. Semantically similar concepts—those with related meanings—trigger stronger interference, as learning one can suppress or displace the other. Conversely, the study identified cases where conceptual knowledge exhibits synergy, with learning one concept actively facilitating the acquisition of related ones. This interplay shows that conceptual knowledge differs significantly in how transferable it is across the model.

Toward smarter training strategies#

Armed with these insights, the team proposed Circuit-aware Experience Replay, a training method that uses circuit topology to identify which concepts are most vulnerable to forgetting and prioritizes retaining them. The approach suggests that understanding the geometry and dynamics of concept learning could enable more effective continual pretraining strategies tailored to prevent catastrophic forgetting while maximizing knowledge transfer.