New benchmark tests whether AI foundation models live up to clinical promises in healthcare data

Foundation models in medicine have generated considerable enthusiasm for their ability to overcome traditional machine learning's dependence on labeled datasets and rigid task design. Researchers from Columbia University Irving Medical Center and collaborators have now systematically tested whether these models actually deliver on those claims by developing an evaluation framework called FoMoH that assesses six state-of-the-art EHR foundation models across clinically relevant scenarios.

The benchmark design#

The evaluation spans 14 prediction tasks covering both patient prognostication and early detection of acute and chronic illnesses. Testing drew on data from more than 6 million patients across Columbia University Irving Medical Center and MIMIC-IV. The researchers went beyond simple discrimination metrics, incorporating rigorous assessment of model calibration (whether predicted probabilities match actual outcomes) and fairness across different patient populations. Their design incorporated explicit controls for data contamination and ensured reproducibility throughout.

What foundation models got right#

Foundation models showed genuine advantages in certain areas. Top performers notably outpaced traditional supervised baselines when working with limited labeled training data—a key promise of the foundation model approach. Additionally, these advanced models demonstrated more equitable performance across diverse socio-medical patient groups compared to conventional approaches, suggesting they may help reduce healthcare disparities.

Significant shortcomings remain#

However, the benchmark surfaced critical limitations. Foundation models struggled in low-prevalence disease settings, potentially because their pretraining objectives inadvertently discard rare clinical signals. When labeled data was sparse, these models also exhibited weaker calibration—meaning their confidence estimates diverged from actual accuracy. Perhaps most concerning for real-world deployment, cross-institutional transportability proved challenging, as models trained at one medical center did not transfer reliably to different healthcare systems.

Clinical implications#

The findings suggest foundation models represent meaningful progress for EHR applications but are far from solving core challenges in medical AI. The benchmark provides a reproducible framework for tracking future improvements and helps clarify which clinical deployment scenarios may be most suitable for these emerging technologies.