New Benchmark Tests AI Models on Multiple Languages and Media Types

A research team has unveiled a comprehensive evaluation benchmark designed to measure multimodal AI systems' ability to follow instructions across different languages and input formats. The new tool addresses a significant gap in how we test large language models that work with speech, images, and text together.

The Problem with Current Benchmarks#

Existing evaluation frameworks for multimodal AI models fall short in several ways. Most focus on English only, test one type of input at a time, rely on short passages rather than longer content, or lack rigorous human review of their results. This piecemeal approach makes it difficult to understand how well these systems truly perform when asked to work across languages and integrate information from multiple sources simultaneously.

Introducing MCIF#

The new benchmark, called MCIF (Multimodal Crosslingual Instruction Following), was built from recordings of scientific talks covering NLP research and related topics. The dataset spans four languages—English, German, Italian, and Chinese—and incorporates three core input types: speech, vision, and text. This parallel design ensures consistent evaluation across all these dimensions.

The benchmark evaluates models on four main task categories:

• Recognition (identifying content)
• Translation (converting between languages)
• Question answering (retrieving specific information)
• Summarization (condensing key points)

What the Testing Revealed#

The researchers benchmarked 23 different models using MCIF and found consistent weaknesses across both modalities and tasks, suggesting considerable room for improvement in how future AI systems handle multimodal, multilingual instruction following.

The benchmark was released under a CC-BY 4.0 open license to encourage continued research and development in this area.