Transformer models show consistent layer-by-layer positional biases that favor different word types

Language models made from transformers exhibit a peculiar weakness: they tend to weight predictions based on where a token appears in the input, not just what it means. A recent paper on arXiv digs into this positional bias by tracing how it develops across the layers of a model, revealing patterns that expose a hidden preference structure inside these systems.

The Problem with Position over Meaning#

Transformer language models show systematic tendencies to favor tokens at particular input positions regardless of whether those tokens are semantically relevant. This positional bias has been observed before—models stumble when handling long contexts and attention patterns reveal position-dependent behavior—but the new research takes a different approach. Instead of looking at external performance or attention weights, the researchers examine how positions actually influence predictions as information flows through each successive layer of the model.

How the Analysis Works#

The researchers developed a layer conductance framework embedded in a sliding-window design, focused on short-context next-word prediction tasks. By isolating the model's internal computations from broader task pressures and context-window effects, they could measure precisely how much each input position contributes to the final prediction at each layer. The resulting positional importance profiles held stable across different texts and even when words were scrambled, indicating these patterns reflect fundamental model properties rather than artifacts of specific datasets.

What Drives the Bias#

The findings expose two competing positional preferences:

• Recency bias strengthens monotonically as you move deeper into the model, meaning later layers increasingly rely on recently seen tokens
• Primacy bias—preference for early tokens—appears subtle and gradually fades through the network

Crucially, this bias is not uniform across all words. The model treats different linguistic categories differently: function words (like "the," "is," "and") show stronger recency bias, while content words (like nouns and verbs) display higher primacy bias. This suggests the model has learned distinct processing strategies for grammatical structure versus semantic information.