Open-Weight LLMs Outperform Rule-Based Systems at Extracting Disease Labels from CT Scans

Researchers have demonstrated that lightweight open-source large language models can automatically extract disease information from chest, abdomen, and pelvis CT reports with performance exceeding both rule-based systems and fine-tuned specialized models.

In a retrospective analysis of over 40,000 CT reports, five different open-weight LLMs were tested zero-shot—meaning they were given no task-specific training—to assign 15 diagnostic labels across three organ systems. The models outperformed a conventional rule-based algorithm and a fine-tuned version of RadBERT, a model pre-trained on radiology text.

Model Performance and Agreement#

The MedGemma variants achieved the strongest inter-model consistency, with a Cohen's kappa coefficient of 0.90 across 12,000 held-out reports. When measured against 1,789 radiologist-annotated labels, Gemma-3 27B reached a macro-averaged F1 score of 0.82, compared to 0.66 for RadBERT and 0.64 for the rule-based approach. A majority-vote ensemble combining multiple models achieved an even higher score of 0.84.

Where Models Struggled#

Performance was notably lower on subjective findings. Kidney lesions and atelectasis (partial lung collapse) proved challenging, with average F1 scores of 0.44 and 0.67 respectively across all models. These conditions require clinical judgment to assess their clinical significance—not just their presence—which may explain the variation.

The Labeling Convention Problem#

Critically, much of the disagreement between models and radiologists stemmed not from errors but from differing interpretation standards. When the researchers relabeled the test set using simpler criteria that disregarded clinical context, scores improved for all LLMs on kidney lesions. For atelectasis, however, only the LLMs benefited; the rule-based and BERT systems actually performed worse, suggesting they applied more clinically-nuanced definitions.

This mismatch reveals a fundamental insight: performance metrics depend heavily on how humans and machines define what counts as a positive finding. When compared against a public dataset with more literal labeling conventions (CT-RATE), all models scored higher than against the original radiologist annotations.

Implications#

The findings suggest that open-weight LLMs are viable alternatives to expensive proprietary models or custom rule-based systems for automating disease extraction from radiology reports. However, successful deployment requires careful alignment between model output and the intended labeling philosophy—whether that prioritizes sensitivity, specificity, or clinical relevance.