Modular Sparse Experts Enable LLMs to Learn Continuously Without Forgetting Old Knowledge

Researchers have introduced Mixtures of SubExperts (MoSEs), a novel approach that allows large language models to continue learning throughout their operational lifetime while sidesteping two fundamental obstacles: forgetting previously acquired knowledge and growing prohibitively large.

The challenge in continual learning comes down to a delicate balancing act. Models must absorb fresh information and adapt to new tasks without erasing what they already know—a problem known as the stability-plasticity dilemma. Additionally, they must do so while keeping their parameter count manageable, since model size directly impacts computational costs.

How MoSEs Work#

MoSEs tackle this puzzle by breaking model capacity into small, reusable building blocks called SubExperts. Rather than retraining the entire network or creating entirely separate models for each new task, these lightweight modules are added to transformer layers. A learned routing mechanism dynamically decides which subset of modules activates for a given input, allowing the model to combine different sub-functions as needed.

This sparse, modular structure creates what the researchers call a "structured decomposition of parameter space." Knowledge stays isolated within particular modules, which cuts down interference between tasks, while the routing system enables the discovered components to recombine for new situations—effectively allowing the model to generalize to unseen combinations of learned skills.

Balancing Three Key Demands#

The framework addresses the core tradeoff through three mechanisms:

• Stability: Activating only a sparse subset of modules for each task keeps old knowledge sheltered from new learning signals.
• Plasticity: The routing-driven composition and selective expansion allow the model to build new capabilities by mixing and extending existing components.
• Scalability: Because only a fraction of parameters activate at any given time, the model's effective capacity grows sublinearly—avoiding the linear blowup that plagues task-isolated approaches.

Empirical Results#

Testing on benchmark datasets (TRACE and SuperNI) demonstrated that MoSEs reduced catastrophic forgetting compared to existing parameter-efficient fine-tuning methods, improved how well skills transfer between related tasks, and achieved better parameter efficiency. The method establishes what researchers call a new "Pareto frontier"—a configuration that excels on multiple dimensions simultaneously: strong performance, constrained parameter budgets, and robust continual learning.