New technique trains compact AI models 1,000x faster by cloning teacher networks efficiently

Creating performant smaller language models typically requires substantial computational resources, even when leveraging knowledge distillation and pruning techniques from larger teacher models. Researchers have now proposed a solution that dramatically reduces training costs while maintaining competitive performance.

The Challenge with Current Approaches#

Existing knowledge distillation methods struggle with three fundamental problems. Hard pruning causes irreversible loss of information from teacher models. Student models often require explicit modules to align their internal representations with those of teachers. Additionally, valuable signals generated by Feed-Forward Networks (FFNs) are frequently overlooked or underutilized during the distillation process.

How Low-Rank Clone Works#

The team introduced Low-Rank Clone (LRC), a pre-training approach designed to create smaller language models that behave nearly identically to their larger teachers. The method trains a collection of low-rank projection matrices that serve dual purposes: they enable soft pruning by compressing teacher weights, and simultaneously align student activations—including FFN outputs—with the teacher's signals. This unified design eliminates the need for separate alignment modules while maximizing knowledge transfer efficiency.

Efficiency Breakthrough#

Testing with open-source teacher models including Llama-3.2-3B-Instruct and Qwen2.5-3B/7B-Instruct variants, LRC achieves performance matching or exceeding state-of-the-art models trained on trillions of tokens, while using only 20 billion tokens. This represents roughly 1,000x greater training efficiency compared to traditional approaches.

The research has made code and model checkpoints publicly available, enabling other researchers to build on these findings.