AI medical diagnosis tools show strong consistency but vulnerable to irrelevant information

A fresh examination of two leading language models' capacity to diagnose medical conditions reveals a mixed picture: while they deliver rock-solid consistency when answering the same question multiple ways, they're surprisingly susceptible to distraction from irrelevant information and make clinically dubious decisions when presented with added context.

The Study#

Researchers tested Google Gemini 2.0 Flash and OpenAI ChatGPT-4o across 52 clinical scenarios, each tweaked under controlled conditions to probe three key vulnerabilities: whether the models stuck to the same conclusion when given equivalent information in different words, how readily they switched diagnoses when fed misleading but plausible narrative details, and whether they appropriately adapted their assessment when clinically relevant new information arrived.

Perfect Consistency, Imperfect Judgment#

Both models proved equally bulletproof on consistency—they delivered identical diagnoses across all equivalent variants and repeated queries with zero deviation. Yet this stability evaporated when the rules changed. When researchers inserted irrelevant but realistic background details into scenarios while keeping core clinical evidence untouched, Gemini shifted its diagnosis in 40% of cases and ChatGPT in 30%.

The divergence widened on contextual awareness. ChatGPT responded to added patient history, lifestyle information, or diagnostic findings more often (77.8% versus 55.6%), but a significantly larger share of its pivots—one-third—lacked clinical justification. Gemini made fewer context-driven changes overall, yet nearly two-thirds of the shifts it did make were deemed appropriate by physician reviewers, compared to just over half for ChatGPT.

Implications for Clinical Use#

The findings underscore a critical gap between machine reliability and clinical safety. Consistency in controlled laboratory conditions failed to guard either model against manipulation through noise or unjustified diagnostic revisions triggered by context shifts. The authors conclude that before these tools can reliably distinguish signal from noise or acknowledge insufficient evidence, deploying them in diagnosis requires sustained clinician oversight and robust structural safeguards.