New Technique Speeds Up Vision Transformer Model Inversion While Preserving Knowledge Transfer

Model inversion—the process of reconstructing synthetic training data from a pretrained neural network without access to the original data—is a key technique in data-free learning. However, applying it to modern Vision Transformers (ViTs) has proven computationally expensive due to their complex self-attention mechanisms.

The Problem with Current Approaches#

Previous work on accelerating model inversion for ViTs, known as Sparse Model Inversion (SMI), attempted to improve efficiency by pruning and removing patches deemed unimportant during the reconstruction process. Researchers behind SMI argued that these discarded patches were actually hindering knowledge transfer.

A new arXiv preprint from Seongsoo Heo and colleagues challenges this assumption through empirical analysis. Their findings reveal that even randomly selected patches can gradually acquire transferable knowledge if allowed to continue evolving during inversion—suggesting that premature pruning actually wastes valuable information.

Patch Rebirth Inversion: A Better Balance#

To address this inefficiency, the team proposes Patch Rebirth Inversion (PRI), a progressive strategy that detaches the most informative patches incrementally while allowing remaining patches to continue learning. Rather than permanently discarding patches, PRI keeps them in the optimization process, enabling less informative patches to gradually accumulate class-relevant knowledge—a phenomenon the researchers term the "Re-Birth effect."

This approach balances two types of knowledge extraction:

• Class-agnostic features — general patterns useful across categories
• Class-specific features — attributes unique to particular data classes

Performance Gains#

Experimental results demonstrate significant improvements over competing methods:

• 10× faster than standard Dense Model Inversion (DMI)
• 2× faster than existing Sparse Model Inversion
• Consistently outperforms SMI in accuracy
• Matches the performance of DMI while using far less computation