AI
New Benchmark Evaluates AI Models' Ability to Extract Data from Noisy Medical Reports
MedStruct-S tests how well language models handle real-world challenges in extracting structured information from OCR-scanned clinical documents.
Extracting structured medical data from patient reports remains a persistent challenge in healthcare AI. A newly published benchmark aims to measure how well modern language models handle this real-world task under practical constraints.
The Challenge of Clinical Data Extraction#
Medical records often arrive as scanned documents with optical character recognition (OCR) applied—a process that introduces errors and inconsistencies. Beyond noise, clinical reports vary widely in how they label key information: the same field might be called "Patient ID," "Patient Identifier," or abbreviated differently across institutions. Current evaluation frameworks don't adequately test how models cope with these variations, making it difficult to assess their reliability in production settings.
What MedStruct-S Offers#
Researchers have introduced MedStruct-S, a benchmark dataset containing 3,582 annotated pages from actual clinical reports. The benchmark targets three interconnected tasks:
- Field discovery — identifying data labels (keys) that appear in reports
- Key-conditioned question answering — retrieving specific information once a field is known
- End-to-end extraction — locating and pairing all keys with their corresponding values
By grounding evaluation in unknown, heterogeneous key representations and OCR-induced noise, the benchmark better reflects the messiness of real healthcare environments.
Model Performance Across Paradigms#
The researchers benchmarked nine language models ranging from 0.11B to 103B parameters, divided into two architectural approaches: encoder-only sequence labeling (like BERT-style models) and decoder-only structured generation (like GPT-style models). Results revealed an unexpected finding: encoder-only models achieved superior performance on non-empty key-conditioned QA tasks, despite being orders of magnitude smaller than their decoder-only counterparts. When models of similar scale were compared directly, encoder-only architectures still generally outperformed their peers. Only when allowing unlimited model size variation did fine-tuned decoder-only models claim the top overall slot.
