Evolutionary Algorithm Makes LLM Compression Faster and More Effective

Compressing large language models to run on resource-constrained hardware remains a critical challenge. Most existing approaches depend on tedious hyperparameter tuning or greedy heuristics that may miss superior solutions. A new paper from arXiv proposes GeLaCo, an evolutionary algorithm designed to automatically explore the space of compression options through intelligent layer merging.

How It Works#

GeLaCo uses a population-based search strategy—similar to genetic algorithms—to evolve candidate compression solutions over multiple generations. Rather than simply removing neurons or weights, the method collapses layers by merging their parameters in a learnable way. The algorithm evaluates candidates using two criteria: how similar the merged layers' behavior is to the originals (measured via residual updates), and how well the compressed model maintains language modeling accuracy (quantified through KL divergence).

Multi-Objective Optimization#

A key innovation is support for multi-objective search, allowing users to balance model size and quality simultaneously. The technique establishes the first Pareto frontier for LLM compression, showing which compression gains are achievable without unacceptable quality loss. This gives practitioners a menu of trade-off options rather than a single fixed outcome.

Results and Impact#

Testing on both foundational and instruction-tuned models shows GeLaCo outperforms existing alternatives on standard metrics like perplexity and generative task evaluation. By automating the compression search process, the method reduces the manual effort required to deploy large models on memory-limited devices, addressing a major bottleneck in making advanced AI systems more accessible.