New method makes language model training faster and cheaper by embedding syntax during pretraining

A team of researchers has proposed a new approach to reduce the computational burden of training large language models, building on recent work that allows BERT-style models to be trained on a single GPU in just one day.

The Problem: Expensive Pretraining#

Pretraining remains one of the most resource-intensive steps in developing language models, creating a barrier for many researchers who lack access to large computing clusters. The Cramming technique (Geiping and Goldstein, 2022) made progress on this challenge, but there remains room for improvement in how training resources are allocated.

A New Angle: Dependency Agreement Integration#

The researchers' innovation, called Dependency Agreement Cramming (DA-Cramming), takes a different tack from previous methods by injecting syntactic information earlier in the training pipeline. Rather than waiting until the finetuning stage to layer in semantic signals, they weave information about dependency agreements — the grammatical relationships between words in a sentence — directly into pretraining itself.

The framework uses a dual-stage workflow with four specialized submodels. These components work together to identify and extract meaningful dependency patterns at the chunk level, then encode these patterns into embeddings that guide the model's foundational learning.

Empirical Results#

Testing across multiple downstream tasks, DA-Cramming substantially outperformed existing cost-effective pretraining methods, suggesting that enriching early-stage training with syntactic structure yields meaningful performance gains without proportionally increasing computational overhead.