Speech LLMs Can Function Effectively With Only 60% of Decoder Layers

Speech-based large language models (SpeechLLMs) pack the majority of their parameters into decoder layers that process encoded audio input. A new study reveals significant portions of this decoder capacity may be superfluous.

The Redundancy Problem#

Decoder components in SpeechLLMs typically consume over 90% of total model parameters, yet researchers find that much of this capacity goes unused. By testing models across multiple scales—ranging from 1 billion to 8 billion parameters—the team discovered that speech and text inputs activate similar redundant blocks, suggesting this inefficiency stems from the underlying pretrained language models themselves.

Layer Pruning and Performance#

When researchers systematically removed decoder layers and measured how models recovered performance, larger models proved more resilient to cuts. Seven to eight billion parameter models maintained solid automatic speech recognition (ASR) performance while retaining only 60% of their original decoder layers. Smaller models showed less tolerance for pruning, though the general principle held across the board. The technique of post-pruning "healing"—allowing models to compensate for removed layers—improved robustness of the compressed versions.

Generalizing Beyond Speech Recognition#

The findings extend beyond basic speech-to-text tasks. When applied to speech translation, the same redundant layers emerged regardless of which speech encoder was used, the target language, or the specific downstream task. This consistency points to a broader structural redundancy within the architecture itself, independent of the immediate application.

Practical Implications#

The research suggests SpeechLLMs could be deployed as single, pruned backbones serving multiple speech-related tasks without sacrificing performance. This approach could substantially reduce computational demands and memory footprint while maintaining capability across diverse applications, making deployment more practical for resource-constrained environments.