Extreme compression boosts LLM inference speed up to 7× on CPUs and 6.7× on GPUs

Compressing large language models to just 2-bit representation—a radical reduction in numerical precision—can now run inference tasks substantially faster while maintaining competitive accuracy, new research demonstrates. This advance makes LLM deployment far more practical for resource-constrained environments and edge devices.

The core innovation#

The work builds 2-bit microkernels from the ground up for modern processors, then integrates them into full LLM inference systems. On CPUs, the researchers achieve performance nearly at theoretical maximum capacity, besting the leading open-source inference runtime by 2.2 times and delivering up to 7× faster execution compared to conventional 16-bit models. GPU experiments show similar gains: the team implements mixed-precision, 2-bit kernels for Intel Xe2 graphics processors and incorporates them into the popular vLLM framework, yielding speedups of up to 6.7× over 16-bit baselines.

Why this matters#

Ultra-low-bit quantized models preserve perplexity and task accuracy nearly as well as their full-precision versions, yet consume far less memory, bandwidth, and energy. Until now, the runtime software supporting these compact models lagged behind their theoretical potential. This research narrows that gap by optimizing the actual compute kernels—the lowest-level code that performs matrix operations—rather than assuming generic libraries would suffice. The result is inference that approaches optimal hardware utilization across both CPU and GPU architectures.

What's ahead#

The findings suggest 2-bit quantization is now practical for production LLM workloads, opening doors to deployment on phones, embedded systems, and cost-sensitive data centers. Further work may push compression even lower or extend these techniques to newer accelerators.