Language models struggle with selective skill removal: forgetting doesn't generalize reliably

Researchers investigating how transformer language models respond to "unlearning" have discovered that the process is far messier than expected. When models are deliberately trained to forget a skill by being shown random labels for task examples, their behavior shifts in ways that are difficult to predict and control.

How the forgetting works#

The study involved fine-tuning language models on randomized labels to make them forget specific tasks. The approach successfully produces near-random predictions on the individual examples used during this "forgetting" phase. However, the critical question—whether this forgetting generalizes to similar unseen examples—yields strikingly inconsistent answers depending on the task.

Generalization varies wildly by task#

The variability is striking. On entailment classification, forgetting generalizes robustly: models stop producing meaningful predictions on new entailment tasks altogether. But for tasks like physical reasoning and science question-answering, the effect is narrow—models learned to randomize only on the specific training examples and continue answering similar new questions correctly.

Intriguingly, task difficulty offers no clues about whether forgetting will generalize. Instead, the researchers found that success correlates (weakly) with:

• Low initial confidence: Models that were less certain about their original predictions before forgetting were more likely to show broad unlearning effects
• Low representation variability: Tasks where the model's internal representations were more uniform showed better generalization of forgetting

An unsettling discovery about specificity#

Perhaps most unexpectedly, the contents of the training set matter far less than intuition suggests. Models trained on random-label examples of science questions forgot that domain entirely—but then began producing nonsensical output on unrelated entailment tasks. This suggests the forgetting mechanism is capturing something more abstract than just domain knowledge.

The shallow nature of forgetting#

Even when forgetting does generalize across examples, the unlearning turns out to be superficial. Linear probes applied to the model's internal representations can still perform the "forgotten" task reliably, indicating that the underlying learned patterns remain intact below the surface.