Large Language Models Struggle to Track Team Understanding in Conversation

A team of researchers has developed a framework that tests whether large language models can understand and track what team members collectively know during conversation—and found significant blind spots.

The Challenge of "Theory of Mind"#

The study examines a key limitation in how LLMs process human communication: their ability to infer mental states and detect misalignments in team understanding. Researchers sought to see if models could not only identify shared mental models (the common understanding a team builds while working together), but also spot where individuals diverge in what they think the group knows.

Two-Step Evaluation Framework#

The research employs a novel approach. First, an LLM analyzes task-oriented dialogue from the Cooperative Remote Search Task (CReST) corpus, annotating which elements contribute to shared team understanding. Second, another LLM then compares these model-generated annotations against both human annotations and gold-standard labels to flag discrepancies.

The study evaluated the framework on six dialogues from the corpus, producing three key outputs:

• A paired dataset of human and LLM annotations
• A reproducible evaluation framework for assessing shared mental model coherence
• Empirical evidence of where LLM-based discrepancy detection succeeds and fails

Where Models Fall Short#

While LLMs handle straightforward annotation tasks reasonably well, the research reveals systematic failures in two critical areas. Models consistently struggle when dialogue requires spatial reasoning—tracking where objects or people are located in a physical space. They also falter when interpreting prosodic cues, such as tone, emphasis, and speech patterns that humans naturally use to signal intent or understanding.

These gaps suggest that despite their fluency with language, current LLMs lack the grounded reasoning and multi-modal understanding needed to fully grasp the nuances of human teamwork.