Chain-of-Thought Speech Translation Relies Mostly on Text, Not Speech Audio

Researchers studying Chain-of-Thought (CoT) prompting in speech-to-text translation systems have found that these approaches largely bypass acoustic signals in favor of text transcriptions, undermining a key assumption about their design.

The Problem with Current Speech Translation#

Traditional Speech-to-Text Translation (S2TT) systems typically work in two steps: first converting spoken audio to text via Automatic Speech Recognition (ASR), then translating that text into another language. This cascaded approach carries significant drawbacks—errors from the first step flow into the second, and valuable information embedded in speech's tone, rhythm, and emphasis (prosody) goes unused.

CoT prompting was introduced as a way around these limitations, with the theory that simultaneously accessing both the original speech and its transcription would help systems avoid error cascades and tap into acoustic cues that text alone cannot capture.

What the Research Reveals#

Through attribution analysis, robustness testing with deliberately corrupted transcripts, and direct measurement of prosody sensitivity, researchers found that CoT systems behave much like older cascaded methods. Despite having access to raw speech, the models overwhelmingly depend on text transcriptions and extract minimal benefit from the audio signal itself.

Improving Speech Awareness#

The team tested simple training techniques to boost acoustic utilization:

• Adding direct speech-to-text training data improved how much the system leveraged speech signals
• Injecting noisy or corrupted transcripts during training forced models to rely more on acoustic information and increased overall robustness

These interventions demonstrated that current architectures can be steered toward better speech integration, but doing so requires explicit design choices during training.

Why This Matters#

The findings suggest that without deliberate architectural changes to prioritize acoustic input, even theoretically sound approaches will default to the path of least resistance—leaning on text. Future S2TT systems will need to be explicitly engineered to incorporate speech signals directly into the translation process, rather than assuming that having access to both modalities will naturally lead to using both.