Comprehensive survey maps efficiency improvements across transformer language models

Transformer-based large language models have dramatically expanded what's possible in natural language processing, but at a steep computational cost. A new survey spanning 312 academic papers takes a systematic look at efficiency improvements across the field, covering everything from how models are trained to how they run in production.

What the survey covers#

The research examines efficiency gains across multiple dimensions. Key areas include data curation strategies that reduce training overhead, model architecture redesign that improves computational efficiency, model compression techniques that shrink models without sacrificing performance, and dynamic inference methods that adapt computational workload based on input complexity.

The survey also considers efficiency throughout the model lifecycle—from initial pre-training and fine-tuning phases through practical deployment strategies like prompt engineering and Retrieval-Augmented Generation (RAG), which can reduce redundant computation by retrieving relevant information rather than generating it from scratch.

Evaluation methodology#

To validate findings across the field, researchers benchmarked more than 30 prominent NLP models against 13 different evaluation standards. This statistical analysis reveals which efficiency strategies actually preserve or improve model capability while reducing resource demands.

Why this matters#

The push for efficiency tackles multiple pressures facing the AI research community: sky-high computational requirements that limit who can train and deploy models, energy consumption that raises operational costs, carbon emissions tied to model training and inference, and financial barriers that restrict access to state-of-the-art systems. By surveying the full landscape of optimization approaches, the work points researchers toward more sustainable practices in building and deploying language models.

The survey was submitted to arXiv in May 2024 and has been revised through September 2026.