New benchmark reveals AI language models still struggle with women's health guidance

Researchers have developed a specialized benchmark to measure how well today's most advanced language models handle women's health information—and the results suggest significant room for improvement. The evaluation, called WHBench (Women's Health Benchmark), tested 22 models across 47 expert-crafted clinical scenarios and found that the highest-performing system achieved only 72.1% accuracy on a comprehensive 23-criterion rubric.

The benchmark design#

WHBench focuses on 10 distinct women's health topics, with scenarios deliberately designed to expose weaknesses that matter clinically. The assessment criteria span a wide range of dimensions: clinical accuracy, completeness, safety, communication quality, instruction following, equity considerations, uncertainty handling, and guideline adherence. Crucially, safety failures carry heavier penalties in scoring, reflecting the stakes of medical guidance.

Across roughly 3,100 scored responses, the evaluation reveals patterns that raw accuracy numbers alone might miss. No single model achieved consistent excellence across all dimensions, and even leading systems showed troubling performance in specific areas.

Key findings#

The research uncovered several failure modes that present genuine risks:

• Outdated medical guidance — models sometimes rely on practices no longer recommended
• Unsafe omissions — important clinical information or warnings left out
• Dosing errors — incorrect medication amounts or frequencies
• Equity blind spots — failure to account for disparities in health outcomes or access

Performance was highly variable even among top-tier models, with substantial differences in how frequently each system produced responses containing harmful errors.

Validation methodology#

The research team employed human experts to validate model responses using a structured rubric, yielding moderate agreement at the individual response level but much stronger consensus when ranking systems overall. This suggests WHBench is reliable for comparing models while also underscoring the critical need for expert oversight before AI systems are deployed in real clinical settings.

A public resource, WHBench aims to help track progress toward safer and more equitable AI systems in women's health.