New Tucker Decomposition Method Cuts Fine-Tuning Parameters by Hundreds While Matching LoRA Performance

A newly proposed fine-tuning technique achieves competitive results with existing parameter-efficient methods while reducing trainable parameters by orders of magnitude. The approach, called LoRA-CRAFT, applies Tucker tensor decomposition to attention weights across multiple transformer layers simultaneously, then trains only small adaptation matrices on top of frozen decomposed factors.

How It Works#

Traditional parameter-efficient fine-tuning (PEFT) methods tend to fall into two categories. Some decompose weight gradient updates layer-by-layer, while others apply decomposition to pre-trained weights but handle each layer independently. LoRA-CRAFT bridges these approaches by treating pre-trained attention weights from multiple layers as a unified 3D tensor. Using Higher-Order SVD (HOSVD), the method decomposes this cross-layer structure and locks all resulting tensor factors. The model then adapts through lightweight transformations applied to each frozen factor matrix—substantially reducing the number of new parameters that require training.

Experimental Results#

Researchers tested LoRA-CRAFT on language understanding tasks using RoBERTa models and on commonsense reasoning benchmarks with LLaMA2-7B and LLaMA3-8B. The findings show that on LLaMA3-8B, the method exceeds the average accuracy of standard LoRA while using hundreds of times fewer parameters. On LLaMA2-7B with a 0.252M parameter budget, the same advantage holds.

Scalability Advantage#

A key observation from the work is that LoRA-CRAFT's efficiency gains appear to compound as models grow larger. The frozen Tucker factors seem to capture increasingly rich cross-layer structure in bigger pre-trained models, making the decomposition more effective. This suggests the approach may be particularly valuable for fine-tuning very large models where parameter overhead becomes a critical concern.