Agentic AI
Survey Maps the Landscape of Low-Precision Training for Large Language Models
A comprehensive review organizes existing methods for training LLMs with reduced numerical precision, categorizing approaches to improve computational efficiency and hardware accessibility.
Researchers have published a systematic survey of low-precision training techniques for large language models, addressing a fragmented research landscape where diverse numerical formats and methods have made it difficult to build unified understanding across the field.
The Challenge of Efficient LLM Training#
Training state-of-the-art language models demands enormous computational resources, creating a substantial bottleneck for accessibility and scalability. Low-precision training—using reduced numerical formats for weights, activations, and gradients—offers a promising path to improve efficiency. Yet the proliferation of different approaches across the community has left the field somewhat disorganized, with researchers lacking a clear, comprehensive map of the terrain.
A Three-Part Framework#
The survey organizes existing methods into three primary categories based on their numerical format foundations:
- Fixed-point and integer-based methods — approaches that represent values as integers or fixed-point numbers
- Floating-point-based methods — techniques relying on reduced-precision floating-point representations
- Customized format-based methods — domain-specific numerical schemes designed for particular hardware or use cases
This categorization reflects how the choice of format directly influences hardware compatibility, computational speed, and practical implementation ease.
Beyond Training Efficiency#
The review extends beyond raw speed gains to examine robustness and deployment reliability under low precision, recognizing that efficiency gains must not come at the cost of model stability or output quality. The authors also explore quantization-aware training, an overlapping area that shares key similarities with low-precision approaches during forward propagation.
The survey concludes by identifying promising research directions to advance the field, with a curated collection of relevant papers made available to the research community.
