Researchers Use Physics-Inspired Optimization to Shrink Large Language Models by Removing Key Blocks

Computer scientists have developed a novel approach to compress large language models by framing the deletion of transformer blocks as a constrained optimization problem rooted in physics. The technique maps the challenge to an Ising glass system, a concept from statistical mechanics, where the energy landscape predicts how well the compressed model will perform on real-world tasks.

How the Method Works#

Rather than simply removing consecutive layers, this optimization framework identifies the best combination of blocks to delete from any position within the model. The approach formulates the problem mathematically, then uses the physical analogy to efficiently rank thousands of candidate configurations. The method requires only forward and backward passes over a calibration dataset, making it computationally tractable even for massive models.

The researchers also show that fast heuristic solvers can approximate the optimal solution in negligible time when exact optimization becomes infeasible—a practical advantage for real-world deployment.

Strong Results on Aggressive Compression#

The technique demonstrates particularly impressive gains in heavy compression scenarios. When reducing Llama-3.3-70B-Instruct by 50%, the method achieved a 23 percentage point improvement over existing block-removal approaches on the MMLU benchmark. For lighter compression tasks, the results match or exceed other state-of-the-art methods across multiple models including Llama-3.1-8B-Instruct, Qwen3-14B, and the recently released NVIDIA-Nemotron-3-Nano-30B model.

Generality Across Architectures#

A key strength is the approach's flexibility—it works on any transformer-based architecture without modification. The team demonstrated this by successfully applying the method to NVIDIA-Nemotron-3-Nano, a model with an unusually complex and heterogeneous block structure combining attention layers and mixture-of-experts blocks. The method outperformed existing techniques on reasoning benchmarks (AIME25 and GPQA) when removing either attention layers or expert blocks.

Why This Matters#

LLM compression addresses a critical challenge: making powerful models deployable on resource-constrained devices. By going beyond simple consecutive pruning and leveraging principled optimization, this work opens new possibilities for creating efficient variants of frontier models without extensive retraining.