New benchmark tackles the challenge of reliably evaluating AI systems that generate multiple types of content

Evaluating whether an AI system has successfully generated images, sounds, or other multimodal content remains deceptively difficult. Automated scoring methods frequently diverge from how humans assess these outputs, particularly when tasks involve generating or combining multiple forms of media. A new research effort addresses this gap by introducing MMMG, a comprehensive benchmark that applies the verifiable-task paradigm to multimodal generation across four different modality combinations: image-only, audio-only, interleaved text and image, and interleaved text and audio.

Why Automatic Evaluation Is Hard#

The challenge in assessing multimodal generation stems from the inherent complexity of diverse outputs. Unlike text, where string matching or semantic similarity can approximate correctness, visual and audio content demands more nuanced judgment. Most automated metrics fail to capture what makes a generated image or sound suitable for its intended purpose, making them unreliable proxies for human satisfaction.

How MMMG Works#

Rather than attempt to automate evaluation entirely, MMMG focuses on tasks that can be verified or nearly verified—either through reference comparisons or by constraining AI judges with detailed evaluation rubrics. The benchmark includes:

• 55 tasks total, with 31 newly developed ones
• 1,288 instructions designed to systematically probe reasoning, controllability, and other essential capabilities
• A hybrid approach combining both automated programs and model-based judgment to keep tasks challenging while enabling reliable assessment

Strong Alignment with Human Judgment#

Extensive validation shows that MMMG achieves an average agreement rate of 94.4% with human evaluations, demonstrating its reliability as a tool for benchmarking.

Performance Insights#

Testing across 29 models reveals telling gaps in current capabilities. The leading model, GPT Image, achieves 70.7% accuracy on image generation—respectable but short of perfection. More critically, performance drops noticeably on interleaved generation tasks that mix text with images or audio. Audio generation emerges as a particular weak spot, suggesting substantial room for improvement in this area and marking it as a priority for future model development.