LLMs Don't Need Their Own Past Responses to Stay Accurate

A new study challenges the standard practice of feeding language models their complete conversation history, revealing that models can achieve comparable performance while using far less context.

Full Context May Be Unnecessary#

Researchers analyzed multi-turn conversations across several leading language models and found that aggressive filtering of dialogue history often preserves response quality. When models replaced all previous assistant outputs with a single-sentence summary—or kept only the latest user-assistant exchange—performance remained roughly equivalent to using the complete conversation transcript. The key finding: this reduced approach used approximately 8 times less context.

Why Models Can Afford to Forget#

The research identified two main reasons for this surprising result. First, roughly one-third of user turns in multi-turn conversations are self-contained questions that don't require prior context to answer. Second, many follow-up questions can be fully addressed by examining only the immediately preceding exchange between user and model, making the broader history redundant.

The Problem With Recycled Responses#

Perhaps most interesting is what the researchers call context pollution—a degradation that occurs when models condition on their own past outputs. Reasoning mistakes, factual hallucinations, and specific writing quirks can cascade across multiple turns when prior assistant responses are included in the prompt. By selectively removing these earlier model-generated passages, the team found they could mitigate this compounding error.

A Smarter Filtering Approach#

Building on these observations, the researchers designed a context-filtering strategy that selectively drops assistant-side history while retaining user prompts and other essential signals. The implications suggest a shift away from the current norm of storing and processing complete dialogue transcripts in favor of keeping only information that directly shapes the next response.

These findings from arXiv (arxiv.org/abs/2602.24287) point toward more efficient conversation systems that could reduce computational overhead while maintaining or even improving response fidelity.