Researchers Develop Predictive Formulas to Guide Language Model Compression

Scaling up language models consistently delivers better performance, but the accompanying explosion in memory and computation costs makes it impractical to deploy them on resource-constrained devices. Pruning—the practice of removing redundant parameters—can shrink these demands substantially, yet determining how much performance will degrade remains largely a guessing game that requires extensive empirical testing.

A Framework for Predictive Pruning#

Researchers have introduced pruning laws, a set of mathematically principled scaling relations that predict how a pruned model will perform based on three factors: its original, unpruned performance; the fraction of parameters removed; and characteristics of the pruning method used. Testing across a range of scenarios—from small 1.3 billion-parameter models to 30 billion-parameter systems and a 20 billion-parameter mixture-of-experts variant—the team evaluated three different pruning strategies (unstructured removal, width reduction, and depth reduction) on eight distinct tasks. The framework achieved an average prediction error below 7%, demonstrating practical reliability.

Key Findings and Applications#

Beyond accurate forecasting, the laws reveal important thresholds: there exist critical pruning points beyond which performance cannot reasonably recover. The functional form of these relations remains consistent across dense and sparse (mixture-of-experts) architectures, as well as across different pruning techniques and models not encountered during development. Task-specific and method-specific coefficients shift in interpretable patterns, meaning practitioners can apply the framework even to new architectures or pruning approaches with minimal additional work.

Practical Impact#

For researchers and practitioners, this work provides a principled pathway to design compression strategies, estimate safe pruning levels without exhaustive hyperparameter sweeps, and deploy models within real-world compute and latency budgets. Rather than running dozens of expensive experiments, teams can now use the scaling relations to rapidly narrow the search space and make confident decisions about how aggressively to compress their systems.