Preserving Raw Conversation Outperforms Structured Memory Extraction for Long-Context LLMs

A team of researchers challenges the conventional wisdom that dialogue systems should compress conversation history into tidy structured summaries. Their findings suggest the opposite: raw verbatim text retrieves information more accurately than carefully distilled artifacts.

The Core Finding#

The researchers ran a careful comparison within an identical retrieval pipeline, changing only what gets stored: either hand-extracted facts and decisions, or unprocessed conversation chunks. Verbatim text won decisively—beating structured extraction by nearly 16 percentage points on the LoCoMo benchmark (43.9% versus 28.0%) and by 22 points on LongMemEval-S (67.4% versus 45.4%). Even adding a semantic graph layer failed to close the accuracy gap.

Why Structure Loses#

The core issue isn't structure itself, but information loss during extraction. When researchers mapped accuracy against how much original dialogue survived the compression process, the correlation was clear: keeping more source text meant better results. The extracted-artifact approach didn't outperform simple RAG-style retrieval overall, though it did show better performance on certain metrics.

Design Implications#

Rather than replacing conversational text with structured summaries, the findings point to a different architecture: artifacts should augment verbatim chunks, not replace them. When both formats coexist in memory, accuracy holds steady. But substituting structured extraction for raw dialogue forfeits the performance advantage.

The research included multiple controls to rule out confounding factors, and the researchers shared code and data for independent verification. Their work is available on arXiv.