Diffusion Models Generate Hard Training Samples Without External Data

Researchers have developed a technique for generating synthetic training data using only the resources already available in a machine learning pipeline. This addresses a common constraint in real-world deployments, where regulations or licensing restrictions often prevent the use of large external datasets or pre-trained foundation models.

How ScoreMix Works#

The method, called ScoreMix, leverages the mathematical properties of diffusion models—a class of generative systems increasingly used in image synthesis. Rather than relying on separate data sources, it blends multiple class-specific signals during the reverse diffusion process. By mixing what the model "knows" about different categories along its generation trajectory, the technique produces domain-tailored samples that challenge the recognition system in useful ways.

The key innovation involves selecting which classes to combine. The team discovered that choosing classes that are far apart in the discriminator's learned embedding space—essentially, classes the recognition model views as very different—generates significantly harder and more informative training examples than choosing similar classes. This counterintuitive finding yielded up to 3 percentage points of additional improvement.

Performance and Practical Advantages#

In testing across 8 face recognition benchmarks, ScoreMix boosted accuracy by as much as 7 percentage points without requiring manual tuning of hyperparameters. The approach proved both robust and practical. Interestingly, the researchers found that the generator's internal conditioning space had minimal influence on the final results, and condition and embedding spaces remain largely uncorrelated under standard alignment metrics.

Self-Contained Data Augmentation#

The technique operates as a closed-loop system: it takes only an existing dataset and the recognition model itself, generating new hard examples to strengthen performance. This self-sufficiency makes it particularly valuable in regulated industries or organizations with strict data governance policies, where importing third-party resources may not be feasible.