MedRepBench: New Benchmark Measures Medical Report Understanding in AI Vision Models

Interpreting medical reports from document images—extracting key findings and translating them into explanations patients can understand—remains a critical gap in clinical AI systems. A new benchmark addresses this challenge head-on by testing how well current vision-language models (VLMs) and large language models (LLMs) handle real-world medical documentation.

The Benchmark#

Researchers have released MedRepBench, a dataset containing 1,925 de-identified medical report images from China covering a range of medical specialties, patient populations, and imaging formats. The focus is narrowly tailored: structured extraction of report elements—fields like test item, measured value, unit, reference range, and abnormality flags—plus generating plain-language explanations grounded directly in what the report states. The benchmark does not attempt to evaluate diagnostic reasoning, treatment planning, or incorporation of patient history.

How It Works#

The evaluation employs two complementary approaches:

• Objective measurement: Field-level recall scoring, tracking how accurately models extract individual structured data points from reports
• Subjective assessment: An LLM-based judge that rates explanations on factuality, clarity, and quality of reasoning using a standardized prompt

The researchers also tested a text-only control setting—feeding high-quality OCR-extracted text to an LLM—to establish an approximate ceiling for performance when character recognition errors are minimized. This reveals how much vision-based end-to-end models still struggle compared to an idealized pipeline.

Improving Performance#

Using the objective metric as a training signal, the team developed a lightweight GRPO-based alignment baseline applied to a mid-sized VLM. This approach yielded field-level recall improvements of up to 6%, demonstrating that targeted fine-tuning can boost structured extraction performance.

Why This Matters#

The analysis exposes practical pitfalls in OCR-plus-LLM workflows, particularly layout-related errors and added latency. These findings underscore the value of robust, end-to-end vision-based approaches for medical report understanding—critical infrastructure for clinical systems that must exchange structured patient information reliably. The dataset and evaluation framework are now publicly available.