New Multimodal Translation Model Combines Audio, Images, and Text Across Languages

Researchers have developed a unified translation system that bypasses traditional cascaded approaches by directly processing audio and visual inputs alongside text. The architecture, called OmniFusion, integrates a multimodal foundation model with a specialized translation engine to enable simultaneous multilingual translation with meaningful latency gains.

The Problem with Cascaded Pipelines#

Current speech translation workflows suffer from a critical bottleneck: they first transcribe spoken words into text using automatic speech recognition, then pass that transcript to a translation model. This two-stage process introduces delay—particularly problematic for simultaneous interpretation scenarios where lag undermines real-time communication. Additionally, cascaded systems ignore visual context (like images) that could help resolve ambiguous or culturally specific language.

While open-source translation large language models have made strides in language coverage and output quality, they remain text-only. Separately, pretrained multimodal foundation models can reason across audio, vision, and text, yet they lack the specialized translation expertise and multilingual breadth needed for high-quality professional translation.

How OmniFusion Works#

The team developed a novel fusion strategy that bridges these two model families. Rather than simply concatenating their outputs, OmniFusion connects hidden states from multiple layers of a multimodal foundation model (Omni 2.5-7B) directly into a translation specialist model (SeedX PPO-7B), allowing joint end-to-end training. This approach lets the translation model leverage both the perceptual capabilities of the multimodal encoder and its own translation expertise in a unified forward pass.

The resulting system can handle three distinct translation tasks:

• Speech-to-text translation
• Speech-and-image-to-text translation
• Text-and-image-to-text translation

Results and Performance Gains#

Testing showed that OmniFusion effectively exploits both audio and visual inputs to improve output quality. Most notably, it reduces latency by approximately 1 second compared to traditional cascaded speech translation pipelines—a meaningful gain for interpreters and real-time communication scenarios. The multimodal fusion approach also improved overall translation quality metrics across evaluated language pairs.