DarwinLM uses evolution to streamline massive language models with 5x less training data

Researchers have introduced DarwinLM, an evolutionary approach to shrinking large language models that achieves state-of-the-art compression results while dramatically reducing the data required for post-compression retraining.

Large language models deliver impressive performance across natural language tasks, but their enormous computational footprint makes deployment challenging in real-world settings, especially where speed matters. Structured pruning—strategically removing parts of the network—can cut model size and improve inference speed directly on any hardware. The challenge is that different layers and components respond differently to being removed, so a one-size-fits-all approach leaves performance on the table.

How DarwinLM Works#

DarwinLM addresses this by combining two key ideas. First, it uses an evolutionary search strategy: the algorithm generates multiple candidate models (offspring) per generation by making targeted changes, then keeps the strongest performers and discards the weaker ones. Second, and critically, it embeds a lightweight retraining process into the search itself, gradually increasing the amount of data used to retrain each candidate in each selection round. Poor-performing models are eliminated early, before they consume excessive computational resources.

Real-World Results#

Testing on Llama-2-7B, Llama-3.1-8B, and Qwen-2.5-14B-Instruct, the method outpaced ShearedLlama—a prior state-of-the-art baseline—while needing just one-fifth of the training data to recover performance after compression. This efficiency gain matters for practitioners with limited compute budgets.

The researchers have made their code publicly available, enabling other teams to build on the approach. Full paper on arXiv.