New Framework Uses Lightweight Masks to Teach Pre-Trained Transformers Multiple Tasks Without Forgetting

Machine learning models often struggle when learning multiple tasks sequentially—they tend to forget earlier knowledge while acquiring new skills, a problem known as catastrophic forgetting. A new paper from arXiv proposes a solution inspired by the lottery ticket hypothesis: a method called Soft-TransFormers that keeps a pre-trained Transformer frozen while adding tiny, task-specific adjustments.

How Soft-TransFormers Work#

Rather than retraining the backbone model itself, Soft-TransFormers applies real-valued multiplicative masks to specific parts of the Transformer's self-attention layers (query, key, value, and output projections). These masks start at one, meaning the system begins exactly at the pre-trained solution. As training proceeds, the masks gradually modulate the pathways within the network, steering the model toward task-specific behavior without overwriting the shared foundation.

The approach is mathematized with convergence and drift guarantees under standard conditions, proving that the masks stay close to their initial values and thus keep the model near its original pre-trained state.

Key Advantages#

• No forgetting: Since the backbone is never modified and each task gets its own mask set, old knowledge is structurally preserved.
• Robust to task-switching errors: Even when the system misidentifies which task is underway, the nearby pre-trained solution means performance degradation is minimal.
• Efficient inference: The overhead is identical to running the unmodified backbone—no computational cost at test time.

Integration and Benchmarks#

Soft-TransFormers can work with existing prompt-based continual learning methods (L2P, DualPrompt, HiDe-Prompt, and NoRGa), selecting masks through task-key matching, entropy gradients, or a learned classifier. Testing on standard class-incremental benchmarks—Split-CIFAR100, Split-ImageNet-R, CUB-200, and 5-Datasets—the framework consistently outperformed prompt-based, adapter-based, and LoRA-style approaches while using comparable numbers of trainable parameters.