Hugging Face brings 4-bit model quantization to transformers with QLoRA method

Hugging Face has integrated 4-bit quantization support into its transformers library, making large language models dramatically more accessible to researchers and practitioners without access to high-end hardware.

The advancement centers on QLoRA (Quantized Low-Rank Adapter), a technique introduced in a new paper by Dettmers et al. that slashes memory requirements while preserving performance. According to the research, users can now fine-tune a 65-billion parameter model on a single 48GB GPU—a task previously requiring far more costly infrastructure.

How it works#

QLoRA combines several innovations:

• 4-bit NormalFloat (NF4): An information-theoretically optimal data type for normally distributed weights
• Double quantization: Quantizes the quantization constants themselves, reducing memory footprint by an additional 0.4 bits per parameter
• Paged optimizers: Manage memory spikes during training

The method compresses model weights to 4-bit storage format while keeping computations in higher precision (typically 16-bit bfloat16). Gradient backpropagation flows only through trainable LoRA layers, not the frozen base model—dramatically cutting memory costs without sacrificing accuracy.

Real-world results#

The Guanaco models, created by fine-tuning LLaMA on the OpenAssistant dataset using QLoRA, reach 99.3% of ChatGPT's performance on the Vicuna benchmark using just 24 hours of single-GPU training. Even smaller models matched or exceeded previous state-of-the-art results on instruction-following tasks.

Testing on consumer hardware showed the approach works: researchers fine-tuned a 13B parameter model on an NVIDIA T4 (16GB), a GPU widely available in cloud environments.

Getting started#

Implementation is straightforward. Loading a model in 4-bit now requires just:

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(, load_in_4bit=, device_map=)