Self-Supervised Pre-training Enables Learning from Noisy Labels Without Clean Data

Neural networks trained on datasets with mislabeled or corrupted labels tend to memorize errors, leading to poor generalization and reduced accuracy. Most existing solutions require researchers to first identify and set aside a manually verified subset of clean labels—a costly and impractical step. A new approach sidesteps this requirement by leveraging unlabeled data strategically.

The Strategy#

The method combines two training phases. First, a feature extractor is pre-trained using self-supervised learning (SSL) directly on the target dataset but without any labels. This unsupervised step teaches the model to recognize meaningful patterns in the raw data. Next, the pre-trained feature extractor undergoes standard supervised training on the same noisy dataset, allowing it to learn task-specific mappings from those learned representations.

The researchers tested both contrastive methods (which learn by comparing similar and dissimilar examples) and non-contrastive approaches, finding that both can boost robustness.

Testing and Results#

Across multiple experiments with both artificially injected and real-world label noise, in-domain SSL pre-training consistently improved classification accuracy and downstream label-error detection. The gains became more pronounced as noise levels increased, suggesting that the approach is particularly valuable in challenging scenarios. At low noise rates, the method matched performance of models pre-trained on large public datasets like ImageNet and DinoV2, but substantially outperformed those transfer-learning baselines when noise was severe.

Why This Matters#

By eliminating the need for a clean labeled subset, this approach opens up practical applications in settings where data quality is unknown or difficult to verify. The technique scales across diverse architectures and tasks, suggesting broad applicability for real-world machine learning pipelines.