New Benchmark Reveals Vision-Language Models Struggle With Temporal Medical Reasoning

Researchers have introduced a new evaluation framework that exposes significant gaps in how large vision-language models (LVLMs) handle medical image analysis—specifically when tracking how a patient's condition evolves over time. The benchmark, called TemMed-Bench, mirrors real clinical workflows in a way that existing medical AI tests do not.

The Gap Between Benchmarks and Clinical Practice#

Most current benchmarks for medical vision-language models focus on a snapshot assessment: analyzing a single image from one patient visit. However, clinical decision-making rarely works that way. In reality, physicians routinely compare earlier scans and examinations with recent ones to detect progression, improvement, or unexpected changes. TemMed-Bench addresses this mismatch by requiring models to reason across temporal sequences of medical images.

How the Benchmark Works#

The evaluation consists of three task types designed to test temporal reasoning:

• Visual question-answering (VQA): Models must answer questions about changes between imaging visits
• Report generation: Systems create clinical narratives describing condition shifts
• Image-pair selection: Models identify which images correspond to specific timepoints or conditions

The benchmark draws from a knowledge corpus of over 17,000 instances grounded in authentic clinical scenarios.

Results Reveal Widespread Limitations#

Testing across twelve LVLMs—both proprietary and open-source models—revealed troubling findings. Most systems failed to meaningfully track patient condition changes when working with temporal sequences. In closed-book settings (without external reference material), a substantial portion of models performed at levels indistinguishable from random guessing. This suggests that existing training approaches have not adequately prepared these systems for real-world medical reasoning tasks.

Retrieval Augmentation Shows Promise#

When researchers supplemented models with retrieved information—combining both visual and textual data from the medical domain—performance improved noticeably. Multi-modal retrieval augmentation outperformed text-only retrieval alone across most models tested, with the VQA task showing an average gain of 2.59%. This finding points toward a practical path for enhancing model capabilities without requiring complete retraining.