Audio-Language Models Struggle to Use Clinical Context for Dysarthric Speech Recognition

Automatic speech recognition remains a significant challenge for people with dysarthria and other speech conditions, even as newer audio-language models promise advances. Researchers have now demonstrated that simply adding clinical context to these systems does not reliably improve their ability to transcribe atypical speech—a surprising limitation for models designed to handle multimodal information.

The Study#

The team created a benchmark using data from the Speech Accessibility Project (SAP) to systematically test whether nine different audio-language models could leverage clinical information. The test varied the type of context provided at inference time: diagnosis labels (such as Parkinson's disease or cerebral palsy), clinician-derived severity ratings, and increasingly detailed clinical descriptions. The goal was straightforward—determine whether richer contextual input led to better transcription accuracy for dysarthric speech.

Prompting Yields Minimal Gains#

The results were sobering. Across all comparisons, models showed negligible improvement when given diagnosis-informed or clinically detailed prompts. In many cases, performance actually got worse—word error rates increased rather than decreased. This finding suggests that current models, despite their technical sophistication, are not genuinely processing and using clinical context the way practitioners might hope.

Fine-tuning Shows Promise#

A different approach proved more effective. When researchers moved beyond simple prompting to fine-tune models using low-rank adaptation (LoRA) across a mixture of clinical prompt formats, they achieved a word error rate of 0.066—a 52% relative improvement over the baseline frozen model. This strategy also maintained performance in scenarios where clinical context was unavailable, suggesting a path forward for more robust and adaptable systems.

Who Benefits Most#

The fine-tuned models showed uneven progress. Speakers with Down syndrome and those with mild-severity speech conditions experienced the most substantial gains, while improvements for other populations were more modest. These subgroup patterns highlight an important challenge: building truly inclusive speech recognition systems requires attention to variation within the disability community.

Implications