New Framework Evaluates How Well AI Systems Handle Photos and Videos in Retrieval Tasks

Researchers have developed MiRAGE, an evaluation framework designed to assess how well retrieval-augmented generation (RAG) systems work when pulling information from multimodal sources—including text, images, audio, and video. As online information increasingly comes in diverse formats, RAG systems need better ways to incorporate and cite these different media types, yet current evaluation methods remain heavily focused on text alone.

The Problem with Text-Only Evaluation#

Traditional RAG assessment tools were built around text documents and don't adapt well to multimedia environments. Existing metrics like ALCE, ARGUE, and RAGAS struggle to properly evaluate systems working with images, audio, or video, creating a gap between how these systems are tested and how they actually perform in real-world applications where multimodal content is increasingly common.

How MiRAGE Works#

The framework uses a claim-centric approach, breaking down generated outputs into distinct factual claims and then evaluating two key dimensions:

• InfoF1: Measures whether the generated content is factually accurate and covers the relevant information from source materials
• CiteF1: Assesses whether claims are properly supported by citations and whether all citations in the output are justified

This two-pronged system provides a more nuanced picture of RAG quality than single-metric approaches.

Performance and Generalization#

When applied by human evaluators, MiRAGE aligned strongly with overall judgments about output quality. The researchers also created an automated version of the framework and compared it against multimodal adaptations of the three existing text-focused metrics. MiRAGE outperformed all three competitors on text-based tasks while being the only approach to successfully generalize to multimodal sources. The team released open-source implementations and provided detailed guidance on evaluating multimodal RAG systems.