Hiding Unreliable AI Predictions Backfires: Clinicians Miss More Diagnoses When AI Stays Silent

Researchers studying human-AI collaboration in clinical settings have uncovered an unintended consequence of a common safeguard: hiding uncertain AI predictions from human decision-makers may reduce one problem while creating another.

The Problem Automation Bias Creates#

When AI systems make mistakes in high-stakes environments like medicine, human operators often trust them anyway—a phenomenon called automation bias. This tendency can amplify errors: even accurate AI models will eventually generate wrong predictions, and clinicians relying on those predictions tend to make poorer decisions than they would on their own.

How Selective Prediction Was Supposed to Help#

To counter this risk, researchers have proposed selective prediction: an AI system refrains from offering guidance when uncertain about its answer, essentially telling the user "I don't know" rather than guessing. The underlying assumption is straightforward: when the AI abstains, users should revert to their normal decision-making without its influence—reversing any harm caused by inaccurate suggestions.

What the Study Actually Found#

A new empirical study from arXiv tested this assumption by asking 259 clinicians to diagnose and treat hospitalized patients under three conditions: without AI, with AI showing all predictions, and with AI using selective prediction. The findings painted a nuanced picture:

• Without AI: clinicians achieved 66% accuracy
• With inaccurate AI shown: accuracy dropped to 56% (a 10-point decline)
• With selective prediction: accuracy recovered to 64% (nearly restoring baseline)

On the surface, selective prediction appeared to work. Yet the researchers discovered that the nature of mistakes shifted dramatically. When clinicians learned the AI was abstaining, they became more conservative: they missed diagnoses 18% more often and failed to recommend treatments 35% more frequently compared to operating without AI at all.

What It Means#

The findings underscore a critical lesson in human-AI system design: good intentions do not guarantee good outcomes. Hiding uncertainty from users may preserve overall accuracy metrics while changing decisions go wrong—in a medical context, missing a diagnosis or skipping a necessary treatment can be as dangerous as acting on a false positive. The research highlights the need to test and validate real-world behavior rather than relying on assumptions about how humans will interact with AI abstractions.