Researchers Boost LLM Speed by Pruning Upper-Layer Attention Heads with Adaptive Rescaling

A team of researchers has introduced a novel approach to compressing large language models by strategically removing redundant attention heads from the network's higher layers, combined with an adaptive rescaling technique that maintains model quality.

The Problem with Current Pruning#

Existing structured pruning methods for LLMs typically rely on heuristics that eliminate attention heads across all layers indiscriminately. This approach fails to account for the structural position of each head within the network, potentially removing components that are critical to model performance in certain layers.

How the Method Works#

The new algorithm focuses pruning efforts on the model's upper layers, where researchers found opportunities for compression without significantly impacting downstream performance. Critically, the team introduced an adaptive rescaling parameter that compensates for changes in token representation magnitude that naturally occur when attention heads are removed. This calibration step helps maintain the semantic quality of the model's internal representations after pruning.

Evaluation Across Multiple Models#

The researchers tested their approach on four major open-source language models:

• LLaMA3.1-8B
• Mistral-7B-v0.3
• Qwen2-7B
• Gemma2-9B

Experiments spanned 27 datasets covering both text generation and classification tasks, offering a broad assessment of the method's effectiveness.

Results and Impact#

The findings consistently showed that this targeted layer-wise approach outperforms existing structured pruning baselines. The improvement was particularly pronounced on generation tasks, where the method achieved notably better results than competing techniques. The work reduces inference latency while maintaining competitive model performance—a key goal for deploying LLMs in resource-constrained environments.