Researchers tackle language interference in multilingual speech AI through targeted distillation

Building speech AI systems that reliably follow instructions across many languages remains a stubborn challenge. Training such models typically demands vast amounts of labeled speech data paired with task-specific annotations—a resource constraint that has limited progress in the field.

The Core Problem#

Recent work has shown promise by using distillation techniques that learn from speech and text using only automatic speech recognition (ASR) data. These approaches compress knowledge from larger text models into smaller speech models by aligning the two modalities through a lightweight projection layer. However, when this strategy expands to support multiple languages in a single model, performance drops significantly. The culprit: language interference—different languages competing and confusing the shared projector that bridges speech and text.

A Language-Aware Solution#

Researchers have developed a new distillation approach that replaces the simple shared projector with a more sophisticated architecture. Their method uses a query bank and a gating network that can select or blend query tokens on a per-language basis, guided by a Q-Former-style projector. This allows the model to maintain separate processing pathways for different languages while learning from shared multilingual speech data, reducing the interference that degrades performance.

Benchmarking the Gains#

Testing on instruction-following tasks shows the method achieves 14% improvement over comparable multilingual baselines. The researchers also created Audio-MLQA, a new multilingual spoken question-answering benchmark synthesized from the existing MLQA dataset using high-quality text-to-speech. Against established speech LLM baselines, their best model demonstrated 32% improvement on this benchmark, suggesting the approach generalizes well across diverse languages and tasks.

Why It Matters#

This work addresses a practical bottleneck in scaling speech AI: the ability to train capable multilingual models without requiring proportionally larger amounts of labeled data. By improving how speech and text representations interact across languages, the technique opens a path toward more accessible instruction-following systems for non-English speakers and underrepresented language communities.