New Training Method Achieves Better LLM Performance With Fewer Trainable Parameters

Researchers have proposed a parameter-efficient training approach that improves large language model performance without increasing model size. The method, called Advantageous Parameter EXpansion Training (APEX), selectively shifts computational resources from underperforming to high-impact parameters during training.

The Core Insight#

Current approaches to strengthen language models typically rely on scaling up total trainable parameters, which drives up computational costs. The new work identifies a more nuanced pattern: certain "advantageous parameters" disproportionately influence model quality, while others contribute less meaningfully. Stronger models tend to possess a higher proportion of these advantageous parameters. APEX exploits this observation by progressively expanding the advantageous parameter set, replacing weaker parameters with additional copies of effective ones.

Performance Gains in Practice#

Experiments across five different base models show substantial efficiency gains. In instruction tuning, APEX matches or exceeds full-parameter fine-tuning while using only 52% of the trainable parameters. In continued pre-training scenarios, the method achieves equivalent perplexity to conventional training using just 30% of the training data, with notable downstream task improvements.

Broader Implications#

The results suggest that parameter quality matters more than sheer quantity. By concentrating learning capacity where it has the most impact, APEX offers a pathway to more computationally efficient model development—a significant consideration as language models grow ever larger.