New mask-based fine-tuning method improves LLM performance without changing weights

A team of researchers has introduced Mask Fine-Tuning (MFT), a novel approach to optimizing large language models that achieves performance improvements through a counterintuitive mechanism: strategic deactivation of model components rather than weight adjustment.

How Mask Fine-Tuning Works#

MFT applies binary masks to already fully-tuned models, effectively toggling neurons and layers on or off. The method uses standard LLM fine-tuning objectives as training signals while keeping the underlying weights frozen. This allows the technique to be layered onto existing well-optimized models without requiring complete retraining.

Performance Gains Across Benchmarks#

Testing on the IFEval benchmark demonstrated measurable improvements: an average gain of 2.70 points on LLaMA2-7B and 4.15 points on LLaMA3.1-8B when MFT was applied to fully fine-tuned baseline models. The gains appeared consistently across different model architectures and application domains, suggesting the approach generalizes well.

Complementary to Existing Optimization#

One notable advantage is that MFT functions as a compatible add-on to other LLM optimization procedures. Rather than replacing existing training methods, it can be deployed alongside them to achieve cumulative improvements. The method also shows compatibility with different backbone architectures, indicating broad applicability.

Expanding Beyond Pruning#

While masking operations have traditionally been associated with network pruning for model compression, this work reframes masking as a broader optimization strategy capable of enhancing overall model capabilities. The research includes detailed ablation studies examining sparse ratios and loss surfaces to understand when and why selective deactivation helps performance.