New technique helps LLMs extend context windows while preserving core capabilities

Expanding the context window of large language models typically involves modifying their positional encodings and running additional training on longer texts. But this approach often comes at a cost: the models lose performance on standard benchmarks designed for shorter inputs.

A new research paper introduces LinearARD, a method aimed at solving this trade-off by restoring model performance after context extension using what researchers call "attention-structure consistency." The work was submitted to arXiv on March 9, 2026.

How the method works#

LinearARD uses a teacher-student framework in which a frozen model with native positional embeddings guides a modified student model. Rather than trying to match internal representations, the technique aligns attention patterns—specifically, it supervises how the model computes relationships between query, key, and value vectors.

The core innovation addresses a practical problem: generating and storing full attention matrices requires memory that grows quadratically with sequence length. To work around this bottleneck, the researchers developed a memory-efficient kernel that computes the mathematical divergence between attention distributions without materializing the entire matrices. It uses summary statistics and recomputes attention scores during the backward pass, keeping memory usage linear.

Performance gains with fewer training tokens#

Testing on LLaMA2-7B scaled from 4,000 to 32,000 tokens of context length, LinearARD recovered 98.3% of the model's original short-text performance while improving on benchmarks that test longer-context understanding.

The efficiency gains are substantial: the method required only 4.25 million training tokens, compared to 256 million tokens needed by comparable baselines like LongReD and standard continual pre-training approaches. Code for the work is available through the researchers' repository.

Why it matters#

Successfully extending context windows without sacrificing baseline capabilities is a practical concern for deploying large language models. This approach demonstrates that attention-based self-distillation, combined with memory-efficient computation, can address the performance degradation problem that often arises during context extension.