LLMs show surprising temporal biases when retrieving repeated information, new study reveals

A new arXiv study reveals that large language models don't just rely on meaning when retrieving information from context—they also exhibit strong temporal preferences that favor certain positions in a sequence.

Researchers tested various pretrained LLMs, including both transformer-based and state-space architectures, by constructing prompts where identical tokens appeared multiple times, with the repeated token surfacing again at the sequence's end. By holding these token positions constant while shuffling everything else, the team removed semantic confounds and isolated purely temporal effects on how models predict the next token.

The Position Preference Pattern#

The findings showed a consistent pattern across model types: while models generally assigned highest probabilities to tokens that followed repeated elements, they displayed a marked bias toward information positioned at the beginning or end of the input. Information embedded in the middle of a prompt proved significantly harder to retrieve reliably, mirroring how human episodic memory retrieves events separated by time.

This temporal bias remained robust even when researchers extended their experiments to include unique semantic contexts with partial overlap, suggesting the effect runs deeper than simple recency or primacy.

Induction Heads Implicated#

An ablation study traced the phenomenon in transformer models to induction heads, a well-known circuit that completes token repetitions. The same temporal preferences appeared across both architectural families—transformers and state-space models alike—indicating this may be a fundamental property of how modern LLMs handle in-context learning.

Why It Matters#

The work refines our understanding of how large language models encode and retrieve information in context, showing that temporal structure shapes retrieval as much as semantic relationships do. These findings offer insight into potential limitations of current architectures when dealing with information positioned at various points in longer sequences.