New system enables real-time speech translation with lightweight, one-way neural design

A team of computer scientists has proposed a new framework for simultaneous speech translation—the task of translating spoken language into text in another language as it's being spoken, rather than waiting for the full audio input to arrive. The challenge has intensified now that large language models show strong translation capabilities, because deploying them to this real-time setting introduces computational bottlenecks and design inflexibilities.

The efficiency problem with existing approaches#

Earlier LLM-based systems for live speech translation either require encoding the same audio multiple times as new chunks arrive—a wasteful overhead—or lock in a fixed schedule for when to pause and generate output. Both strategies cap performance and responsiveness.

EASiST's solution: unidirectional flow and adaptive policy#

The proposed system, called Efficient and Adaptive Simultaneous Speech Translation (EASiST), rethinks the architecture from the ground up. It uses a fully one-way (unidirectional) design for both the speech encoder and the underlying language model. Rather than treating simultaneous translation as a separate problem, EASiST reframes it as an interleaved generation task in which explicit tokens signal whether to read more audio or write translation output.

The system includes a small, trainable policy head—a lightweight neural component—that learns to predict on the fly when to read or write, allowing the model to adjust its behavior based on what it sees in the incoming stream. This replaces the rigid, predetermined schedules of earlier work.

Training and validation#

EASiST employs a multi-stage training routine that first aligns the speech and text representations, then refines both the translation quality and the policy decisions. The researchers also introduce a multi-latency data curation strategy to generate training samples that are semantically coherent across the time-stepped translation process.

Evaluation on both familiar (MuST-C) and out-of-domain (Europarl-ST) English-to-German and English-to-Spanish datasets shows that EASiST achieves better latency-quality trade-offs than several strong baseline methods, suggesting a practical path forward for real-world live translation systems.