Fine-tuning LLMs to 1.58bit: extreme quantization techniques now practical

Hugging Face has published a guide to fine-tuning large language models using BitNet's extreme quantization approach, making the technique accessible beyond well-funded institutions that can pre-train models from scratch.

What is BitNet?#

BitNet, an architecture from Microsoft Research, represents each parameter using just three values: −1, 0, and 1. This delivers extreme quantization of 1.58 bits per parameter—compared to LLaMA's 16-bit floating-point representation. The approach uses INT8 addition for matrix multiplication, theoretically reducing energy consumption by 71.4× for arithmetic operations.

The catch: the original technique required training models from scratch, a costly undertaking few organizations can afford.

Adapting Pre-trained Models#

The researchers successfully fine-tuned a Llama3 8B model using BitNet architecture, releasing three models under the HF1BitLLM organization. Two were fine-tuned on 10 billion tokens; one on 100 billion. Notably, these models outperform Llama 1 7B on MMLU benchmarks.

Using the models with Transformers requires no API changes:

model = AutoModelForCausalLM.from_pretrained(
    "HF1BitLLM/Llama3-8B-1.58-100B-tokens",
    device_map="cuda",
    torch_dtype=torch.bfloat16
)

Key Technical Challenges#

The main obstacle: weight quantization uses a non-differentiable rounding function. The solution employs a Straight-Through Estimator (STE), which lets gradients flow through the rounding operation by approximating its gradient as 1.

During training, the model applies:

• Ternary weight quantization (−1, 0, 1) via symmetric per-tensor scaling
• 8-bit activation quantization using absolute maximum per-token scaling
• Layer normalization before activation quantization to maintain output variance

A critical discovery: randomly initialized BitLinear weights overwhelmed the pre-trained model, causing it to lose prior knowledge. The team addressed this by gradually introducing quantization: