Study reveals why local attention improves transformer performance beyond efficiency gains

A new research paper from arXiv provides formal evidence for why constraining transformer attention mechanisms to local token windows often produces better-performing models, even though the change was primarily adopted for computational efficiency.

The core finding#

Researchers characterized the expressive power of local attention — where each token only looks back at a bounded window of predecessors — versus global attention, which considers all preceding tokens. The team proved that local attention introduces new temporal operators that global attention alone cannot capture, allowing transformers to recognize a strictly larger class of regular languages. Crucially, global and local attention are expressively complementary: neither dominates the other, and combining both yields the most powerful variant.

How expressivity maps to language understanding#

The work builds on prior theoretical results linking fixed-precision transformers with global attention to a fragment of linear temporal logic. By adding local attention, the researchers demonstrated that an additional temporal operator emerges, enlarging the set of patterns the model can formally recognize. This theoretical insight helps explain a practical mystery: why does restricting what tokens can attend to each other sometimes improve model quality?

Validating theory with experiments#

The team tested their predictions on two fronts: formal language recognition tasks and natural language modeling benchmarks. In both settings, hybrid transformers combining global and local attention outperformed models using global attention alone, aligning with the theoretical prediction that the richer expressivity class should enable stronger performance.

Why this matters#

The findings suggest that attention scope constraints aren't merely efficiency tricks—they fundamentally alter what kinds of dependencies a transformer can learn. Understanding these expressive boundaries could guide architecture design beyond transformer models and help explain when and why certain architectural choices improve model capabilities.