Researchers develop metrics to detect how LLMs drift in multi-turn conversations

Large language models operating in complex pipelines—pulling from retrieval systems, memory buffers, tools, and other agents—gradually lose coherence as information flows from one turn to the next. Components may each function normally in isolation, yet critical context slips through the cracks, gets compressed, or routes incorrectly. Existing evaluation methods miss this because they measure individual prompts, responses, or final task success, not the relationship between what a model receives and what it produces next.

Measuring the gap#

Researchers propose treating the chain of prompt → response → next prompt as the core analytical unit, rather than examining pieces in isolation. They introduce structural communication coherence, quantified through two metrics:

• Communication closure assesses whether information the pipeline returns at one conversational turn aligns with what the system encounters in the next
• Normalized conditional action contribution measures whether a message sent actually resolves or clarifies the subsequent reply

Findings across dialogue types#

Analysis spanning 2,171 human-to-human exchanges, 58 human-to-LLM dialogues, and 8 LLM-to-LLM conversations reveals directional interaction patterns. A crucial validation: when researchers swapped a response for one from a different conversation turn while keeping the surrounding prompts unchanged, the measured contribution dropped by 87–92%. This sharp decline demonstrates that the metrics genuinely capture the relationship between turns, not mere coincidence.

No labels required#

A key advantage of this approach: it requires no manually labeled data, reference benchmarks, or hand-coded rules. The framework operates directly on raw operational traffic—actual prompts and responses flowing through production systems—making it possible to define and measure drift without waiting for eventual task failures to surface problems. The next research phase will determine whether these metrics can serve as early warning systems in real deployments.