How speech and text share computation differently across AI models

Researchers have developed a new method to understand how multilingual speech-text AI models align knowledge between spoken and written language. The work exposes fundamental differences in how neural networks organize language processing across modalities—and suggests that current models take vastly different computational approaches.

The core question#

Multilingual AI systems that handle both speech and text rely on some form of cross-modal language alignment, but it was unclear whether models were sharing the same underlying computations for a language regardless of input form, or whether speech and text each get processed by specialized neural circuits. This distinction matters for understanding model robustness, transferability, and design choices.

A generation-step-aware framework#

The researchers introduced a technique that (i) identifies which neurons are selective for particular languages in each modality at different stages of text generation, (ii) separates these neurons into those handling language representation versus language control, and (iii) measures how much neurons overlap between speech and text inputs.

Applying this framework to SeamlessM4T (a prominent multilingual speech-to-text model) and Qwen2-Audio (a decoder-only model), they uncovered striking contrasts:

• SeamlessM4T shows dramatic specialization: only 5–7% of language-representation neurons are shared across modalities. Overlap shifts partway through decoding, with later steps favoring typologically related languages over same-language pairs.

• Qwen2-Audio, by contrast, maintains 45–47% cross-modal sharing and exhibits stable, language-specific alignment that persists throughout generation.

Notably, language-control neurons (those active in later decoding steps) show the opposite trend: they display progressively stronger speech-to-text transfer in SeamlessM4T.

Why it matters#

These results demonstrate that multilingual alignment is not a fixed property but a generation-aware, model-specific phenomenon. The findings suggest researchers need to account for decoding stage when analyzing cross-modal computation, and that architectural choices lead to fundamentally different division of labor between modalities.