Small amounts of speech data dramatically improve LLM-based speech recognition in new domains

Recent advances in automatic speech recognition have shifted toward LLM-based architectures that connect a speech encoder to a large language model through a projection module. While this design allows systems to adapt using text data alone, a critical problem emerges: the language model never experiences the actual noisy signal that comes from the speech processing pipeline, creating a mismatch between training and real-world conditions.

The modality gap problem#

Traditional ASR systems train on paired speech-text data, so the language model learns to expect inputs formatted exactly as the speech encoder produces them. LLM-based systems break this assumption by enabling text-only adaptation, where the model is fine-tuned on transcribed text without seeing corresponding audio. This creates what researchers call a "modality gap"—the language model doesn't understand the actual noise and distortions embedded in real speech encoder outputs.

Testing three adaptation strategies#

A team of researchers set out to investigate whether adding limited speech data could close this gap. They compared three approaches:

• Text-only adaptation: Traditional fine-tuning using transcripts alone
• Paired speech-text adaptation: Using both audio and transcripts together
• Mixed batching (MB): Combining text-only and speech-text examples in the same training batches

Modest audio yields major gains#

The key finding is striking: systems trained with mixed batching using just 10% of the target-domain audio (less than 4 hours of speech) achieved word error rates matching or surpassing systems fine-tuned on the entire dataset using conventional ASR methods. This suggests that even small amounts of real audio provide a powerful alignment signal, teaching the language model how to interpret the speech encoder's outputs without requiring massive amounts of labeled audio.

The approach was tested across both in-domain and out-of-domain settings, and the pattern held consistently—limited speech proved more effective than text-alone training.

Why this matters#

This result has practical implications for deploying ASR systems in new domains or languages where large labeled audio collections are expensive to create. Organizations can now collect a small sample of authentic speech (4 hours represents roughly one workday of continuous audio) rather than investing in comprehensive datasets. The research demonstrates that strategic mixing of modalities, rather than choosing one over the other, offers the most efficient path to domain-specific recognition.