New Method Preserves Rare Expert Knowledge During Fine-Tuning of Mixture-of-Experts Models

Mixture-of-Experts (MoE) models have demonstrated strong performance across various benchmarks, but fine-tuning them through supervised learning presents a significant challenge. The router layers that direct input to specialized experts become unstable during this training phase, often collapsing in ways that harm overall model behavior.

The Problem with Current Approaches#

Existing techniques like DenseMixer and ESFT attempt to address this instability by either mixing information from multiple experts or adding auxiliary losses that encourage balanced activation across all experts. However, these methods introduce noisy gradient signals that frequently degrade model performance rather than improve it.

Through systematic experimentation, researchers discovered an important insight: while certain "super experts" receive far more activation than others, completely removing the less frequently used experts still causes meaningful performance loss. This suggests that even rarely activated experts contain valuable knowledge that contributes to downstream task performance.

A New Framework for Expert Preservation#

Rather than forcing all experts to activate equally, the new approach takes a more nuanced path. The method uses bias-driven sparsification to encourage task-relevant experts to remain highly active while allowing less central experts to become inactive—but crucially, it doesn't eliminate them entirely.

The key innovation involves introducing "condenser experts" that stay permanently active throughout processing. These learnable condenser experts serve as a persistent information pathway, preventing gradient starvation that would otherwise leave sparse experts without learning signals. They essentially allow the model to consolidate knowledge that would fragment across rarely-activated components of the network.

Results and Impact#

Testing on large-scale MoE models showed measurable improvements over existing fine-tuning methods. The framework achieved average performance gains of 2.5% or higher on two important benchmarks: mathematical reasoning tasks and commonsense question-answering. Unlike competing approaches, this method accomplishes these gains without requiring auxiliary loss functions, simplifying both implementation and training.

The analysis indicates that this design more effectively preserves the useful information stored within long-tailed experts under conditions of sparse routing, suggesting a more principled approach to maintaining model knowledge during fine-tuning.