New Method Achieves True Machine Unlearning by Erasing Features at the Source

A new study reveals that most machine unlearning methods fail at their core purpose: they don't actually erase information from neural networks, they just hide it. Researchers tested 14 widely-used unlearning techniques and found that a simple linear transformation could recover forgotten data in seconds, restoring accuracy to within a few percent of the original model.

The Unlearning Problem#

Machine unlearning aims to remove a model's knowledge of specific data—a critical capability for privacy protection and regulatory compliance. However, current evaluation practices focus only on surface-level behavior: what the classifier outputs, accuracy on forgotten data, and confidence scores. This overlooked a fundamental vulnerability.

The team introduced Feature Mapping Recovery (FM-recovery), a diagnostic that exposes how existing methods work. A single linear map, trained on held-out calibration data without access to the forgotten examples, could reverse the unlearning process entirely. The recovered features even enabled pixel-level image reconstruction through a generic decoder. This pattern was consistent across all tested methods: they apply an invertible linear distortion that hides information from one prediction head rather than erasing it from the model's internal representations.

One-Point Contraction: A New Approach#

To address this flaw, researchers propose One-Point Contraction (OPC), an unlearning objective that collapses forgotten-set features to the origin—essentially the zero point in the network's representation space—while preserving the geometry of retained data. The approach has a elegant theoretical property: driving feature representations to collapse is mathematically equivalent to pushing the predictive distribution toward maximum entropy, achieving both behavioral forgetting and representation-level erasure simultaneously.

When a forgotten query reaches the network, it lands in a region treated as out-of-distribution, causing the gradient signal on those queries to collapse along with their features. This differs fundamentally from existing methods' approach of hiding information through reversible transformations.

Performance and Resilience#

OPC stands as the sole method in the benchmark that withstands FM-recovery attacks, resists relearning attempts, and defends against gradient-inversion exploits. Notably, it accomplishes this without degrading performance on retained or test data—a significant advantage over techniques that trade privacy for accuracy.