Punctuation and "Filler" Words Play Outsized Role in How Transformers Store Memory

Researchers have discovered that seemingly trivial tokens such as punctuation marks and articles contribute far more to how transformer-based language models encode and retrieve context than previously assumed.

The Discovery#

A study presented on arXiv reveals that when stopwords, articles, and commas are stripped from inputs, performance drops significantly on benchmark tasks like MMLU and BABILong-4k, even when only truly superfluous tokens are removed. This suggests that what many treat as padding or noise actually serves a crucial function in the model's ability to maintain context over long sequences.

The Mechanism#

The research identifies a strong connection between contextualization—how meaningfully a model encodes information—and linearity, a measure of whether transformations between layers can be approximated by simple linear operations. This relationship hints at how models organize their internal representations around these filler elements, using them as structural anchors for managing contextual information.

The Tool#

To further investigate, the team introduced LLM-Microscope, an open-source toolkit designed to peer into transformer internals. The tool can quantify token-level nonlinearity, assess contextual memory retention, visualize which layers contribute most to output predictions (via an adapted Logit Lens approach), and measure the intrinsic dimensionality of learned representations. These capabilities allow researchers to map precisely how individual tokens shape what a model remembers and uses from its input.

Why It Matters#

The findings challenge the intuition that language models operate like humans reading text—where articles and commas can be glossed over without much loss. Instead, transformers appear to have woven these minor tokens into the fabric of their memory systems, making them essential to long-range understanding.