New Benchmark Exposes How AI Models Lose Information When Converting Between Images and Text

Unified AI models promise to both understand images and generate them from descriptions—capabilities that should theoretically work together. Yet researchers have discovered that even models scoring well on standard benchmarks can fail dramatically when forced to alternate between these two tasks, losing critical details in the process.

In a new study posted to arXiv, researchers introduced the Semantic Drift Protocol (SDP), inspired by the telephone game, to measure how well models preserve meaning when cycling between image-to-text and text-to-image conversion. Starting from an initial image or caption, they repeat this back-and-forth transformation across multiple generations and track what information survives.

How the Test Works#

The protocol mirrors the telephone game where a message deteriorates as it passes from person to person. Researchers began with either an image or caption, converted it to its opposite form, then back again—repeating this process multiple times. They measured semantic preservation using two metrics: Mean Cumulative Drift (MCD), an embedding-based measure tracking content retention across representation spaces, and Multi-Generation GenEval (MGG), which extends existing scoring methods across multiple generations.

To stress-test models beyond typical datasets, the team created a benchmark of 400 image-text pairs from NoCaps and DOCCI, which emphasize novel objects and detailed descriptions rather than the simpler images found in standard evaluation sets like COCO.

What Models Fail and Why#

Testing seven models revealed striking differences. BAGEL maintained high semantic fidelity across multiple generations, while VILA-U and Janus variants collapsed within five cycles despite posting comparable scores on traditional single-pass tests. The researchers identified six recurring failure modes and found that degradation is typically catastrophic rather than gradual—once a critical error occurs, subsequent generations compound the problem.

Why This Matters#

Standard benchmarks evaluate image understanding and generation in isolation using metrics like FID and GenEval for image quality, or MME and MMBench for text understanding. This separation misses a crucial test: whether a model's understanding actually aligns with what it generates. The SDP benchmark exposes these misalignments, revealing that models can lose entities, attributes, spatial relationships, and object counts when their capabilities are combined—failures that single-pass benchmarks never detect.