New System Lets Users Fix Speaker ID Errors in Meeting Transcripts with Real-Time Feedback

The Problem with Automated Meeting Transcription#

Most automatic speech recognition systems that transcribe meetings operate in isolation—they identify who said what without any ability to learn from user corrections. This "open loop" approach means errors in speaker identification persist unchecked, even when the mistakes are obvious to someone who was actually present.

A Human-in-the-Loop Alternative#

Researchers from arXiv have built an interactive system that changes this dynamic. Their approach combines automatic speech recognition and speaker diarization (the process of determining which speaker is talking at each moment) with large language models and human feedback loops.

Here's how it works:

• Initial processing: The system runs live ASR and speaker diarization as participants speak
• Smart summaries: An LLM generates concise summaries highlighting the most important speaker identification errors
• User correction: Meeting attendees can quickly flag mistakes with brief corrective feedback
• Online learning: The system updates the speaker-attributed transcript and enrolls new speaker profiles based on each correction

Handling Uncertainty at Every Step#

The researchers developed mechanisms to account for the fact that errors can occur in multiple places: the initial speech processing, the LLM analysis, and even in user feedback itself. These safeguards help the system correctly interpret what users actually meant to fix, even when signals are ambiguous.

Impressive Real-World Results#

When tested on the AMI headset meeting dataset, the system delivered substantial improvements over a streaming baseline (Google ASR plus ECAPA speaker recognition):

• Diarization error rate dropped by 31.99%
• Speaker substitution error decreased by 52.68%

A pilot usability study suggested additional refinements could further enhance the user experience.