New Model-Blending Technique Boosts LLM Performance Without Expensive Retraining

Training large language models demands enormous computational resources and time-consuming optimization. A team of researchers has developed a more efficient alternative: SoCE, a technique that merges multiple existing models to unlock better performance without the cost of retraining from scratch.

The Problem with Existing Approaches#

Model souping—averaging the weights of multiple models with identical architecture—has emerged as a way to improve LLM capabilities after training. However, previous methods using uniform averaging have a critical weakness: they can degrade precision-dependent tasks, causing certain capabilities to collapse. This shortcoming limits their practical usefulness in production systems.

How SoCE Works#

Rather than treating all models the same, SoCE takes a more nuanced approach. The method begins by analyzing benchmark data to identify which model candidates excel at specific types of tasks. Crucially, the researchers observed that different clusters of benchmark problems often show low correlation with one another—meaning no single model performs best across all categories.

The technique then designates specialized "expert" models for each weakly-correlated cluster and combines them using carefully optimized, non-uniform weights. This targeted blending preserves performance across diverse capability areas while avoiding the averaging-induced collapse that simpler approaches suffer.

Strong Results#

When tested, SoCE delivered measurable improvements in both performance and robustness across multiple domains. The method achieved state-of-the-art rankings on the Berkeley Function Calling Leaderboard, a competitive benchmark for evaluating how well language models can invoke functions and interact with external tools.

The work, submitted to arXiv in November 2025, suggests that smarter composition strategies can rival—or exceed—the gains from expensive, full-scale retraining.