New Model Enables Real-Time Speech Translation Without Language-Specific Alignment Data

Researchers have developed a novel approach to simultaneous speech translation that sidesteps the traditional requirement for expensive, language-specific alignment data. The new system, called Hibiki-Zero, achieves real-time translation while handling the inherent complexity of converting speech across languages with different grammatical structures.

The Problem with Traditional Approaches#

Building systems that translate speech in real-time has historically depended on word-level alignments between source and target languages—essentially, explicit mappings showing which words correspond to which translations. Creating such alignments at scale has proven costly and labor-intensive. Researchers typically resort to synthetic alignments generated through language-specific heuristics, which often produce suboptimal results and limit scalability across diverse language pairs.

How Hibiki-Zero Works#

The model takes a two-stage training approach. First, it learns to perform speech translation using only sentence-level aligned data—a much simpler requirement than word-level precision. This initial training establishes a foundation for accurate translation, albeit at higher latency.

In the second stage, the system applies a specialized reinforcement learning strategy using GRPO (presumably a variant of policy optimization) to optimize latency while maintaining translation quality. This decoupling of accuracy training from latency optimization allows the model to balance real-time performance with fidelity.

Performance and Scalability#

Testing across five language pairs translating into English, Hibiki-Zero demonstrates competitive results in translation accuracy, latency, voice transfer, and naturalness. The model also proves adaptable: adding support for a new input language requires less than 1,000 hours of speech data—a significant reduction compared to traditional supervised approaches.

The team has released model weights, inference code, and a 45-hour multilingual evaluation benchmark to support further research in the field.