MetaTT: A Tensor-Train Adapter That Dramatically Shrinks Fine-Tuning Parameters

A new adapter framework called MetaTT offers a way to fine-tune large transformer models while keeping the number of trainable parameters far smaller than traditional methods. Rather than expanding parameters as the product of multiple dimensions, the approach uses a shared Tensor Train (TT) decomposition to scale parameters as a sum—a crucial difference that yields dramatically leaner models.

How MetaTT Works#

MetaTT uses a single shared Tensor Train to compress transformer sub-modules by factorizing across key structural dimensions: layer depth, matrix type, and optionally heads and task information. This design choice avoids the multiplicative explosion that plagues naive parameterization, resulting in substantially fewer parameters to update during adaptation. The framework lets practitioners tune layer-specific and matrix-type variation without ballooning memory or compute overhead.

Benchmark Results#

Tests against LoRA and other recent matrix and tensor decomposition methods show MetaTT achieving competitive parameter-efficiency-to-accuracy trade-offs on standard single-task language modeling benchmarks. The approach proves especially strong in multi-task settings, where it continues to perform on par with state-of-the-art alternatives. This suggests the shared TT structure is robust across diverse downstream tasks.

A Physics-Inspired Optimization Strategy#

The research leverages techniques from many-body physics—specifically the DMRG (Density Matrix Renormalization Group) method—to design a rank-adaptive optimizer. When combined with AdamW, this optimizer improved training outcomes for a target rank, demonstrating that principles from quantum physics can meaningfully enhance deep learning optimization.

The work is presented as a research paper on arXiv, with updates through July 2026.