New Dataset and Training Method Boost Medical AI's Ability to Reason Across Multiple Images

Multimodal AI systems have made significant strides in interpreting medical images, but most current approaches focus on analyzing single images in isolation. In clinical practice, however, doctors routinely need to compare multiple scans—whether different imaging modalities, anatomical views, or images taken over time—to reach accurate diagnoses. A new research paper tackles this limitation by constructing a large-scale training dataset and developing a specialized training method for multi-image medical reasoning.

Building a Comprehensive Multi-Image Dataset#

The team created PMC-MI, a dataset containing nearly 235,000 instruction examples drawn from compound figures found in published biomedical literature. The collection includes over 10,500 examples specifically structured for reinforcement learning and vetted by medical professionals. They also developed PMC-MI-Bench, a manually reviewed evaluation benchmark that separates test examples at the source-article level to prevent data leakage.

This approach leverages an underutilized resource: the richly annotated figures that already appear in medical journals, where researchers often present multiple related images together to support their findings.

A Three-Stage Training Approach#

The researchers introduced M3LLM, a multimodal language model trained using a tailored three-stage pipeline:

• Stage 1: Supervised fine-tuning on multi-image instruction pairs
• Stage 2: Selection-aware reinforcement learning that teaches the model to identify which images or image regions provide the most relevant evidence for answering specific questions
• Stage 3: Supervised consolidation across a broader mix of instruction types

This progression mimics how clinicians gradually build judgment: first learning to process multiple images, then learning to prioritize diagnostic clues, and finally integrating diverse reasoning patterns.

Strong Performance Across Multiple Benchmarks#

Evaluation on PMC-MI-Bench showed M3LLM substantially outperforming comparable models. On three task types, it achieved scores of 81.6, 84.6, and 85.2 compared to baseline scores of 75.6, 79.1, and 77.6. The model also set new records on two publicly available medical benchmarks, achieving 88.6% on OmniMedVQA and 68.0% on MMMU-Med.