New purification method in CLIP's latent space boosts adversarial robustness for zero-shot image classification

Researchers have proposed a novel approach to defending zero-shot image classifiers against adversarial attacks by leveraging CLIP's latent space. The method, called CLIPure, sidesteps the limitations of traditional adversarial training and can theoretically handle unforeseen attack types.

The Challenge of Adversarial Robustness#

Building image classifiers that remain reliable when facing carefully crafted adversarial perturbations remains a significant hurdle in computer vision. Conventional defense strategies often require models to be trained against specific attack patterns, making them vulnerable to novel threats. Zero-shot classifiers—which can recognize categories without task-specific training—present an additional challenge for robust design.

How CLIPure Works#

Rather than relying on attack-specific training, CLIPure pursues a purification strategy: transforming adversarially corrupted images to remove perturbations before classification. The team formulated this challenge using stochastic differential equations, treating purification as a process that mirrors the inverse of the attack mechanism itself. Their theoretical framework models purification risk as the KL divergence between the distributions of denoising and perturbation processes.

The researchers developed two variants operating in CLIP's multi-modal latent space:

• CLIPure-Diff: Uses the DiffusionPrior module from DALL-E 2 to model how CLIP's latent vectors are generated
• CLIPure-Cos: Employs cosine similarity between image embeddings and text descriptions to assess likelihood

Notably, CLIPure-Cos represents the first purification-based defense that doesn't depend on generative models, significantly improving computational efficiency.

Experimental Results#

Testing across multiple benchmarks revealed substantial improvements over prior approaches:

• CIFAR-10: Robustness jumped from 71.7% to 91.1%
• ImageNet: Performance rose from 59.6% to 72.6%
• 13-dataset benchmark: 108% relative improvement in average robustness over previous state-of-the-art methods

These results suggest CLIPure addresses a meaningful gap in making vision-language models more resistant to adversarial manipulation.