New framework balances adversarial robustness and generalization in vision-language models

Vision-language foundation models like CLIP can be fooled by adversarial perturbations while maintaining strong performance on new tasks. A team of researchers has now identified fundamental causes of this vulnerability and developed a solution called Text-Image Mutual Awareness (TIMA) to address them.

The core problem#

Existing approaches to making CLIP more robust against adversarial attacks often come at a cost: they sacrifice the model's ability to generalize to new data. The researchers uncovered three specific reasons why this tradeoff occurs, examining how the model's internal confidence margins and semantic knowledge interact under attack.

Key findings#

The team's analysis revealed:

• Margin inconsistency: Logit margins—the confidence gaps between correct and incorrect predictions—shift predictably between small and large perturbations, suggesting that this pattern can be exploited to improve defenses.

• Semantic underutilization: A strong negative correlation exists between how confident the model is and the semantic similarity between different classes. This means the model isn't fully leveraging its understanding of how concepts relate to one another.

• Semantic degradation: Current methods that modify text embeddings to improve robustness inadvertently destroy the semantic consistency that the pre-trained model learned, weakening generalization.

The TIMA solution#

Rather than choosing between robustness and generalization, the framework operates on both components simultaneously:

The Text-Aware Image tuning module uses an Adaptive Semantic-Aware Margin technique to explicitly recalibrate prediction confidence margins in a way that accounts for semantic relationships between classes.

The Image-Aware Text tuning module applies Semantic Consistent Minimum Hyperspherical Energy to refine text representations while preserving the original semantic structure built into the model during pre-training.

What this means#

Comprehensive testing showed that TIMA outperforms existing defenses by directly addressing the identified technical failures. The approach points toward a path where foundation models can become reliably robust against adversarial attacks without losing the broad generalization capabilities that make them valuable.