New Benchmark Exposes Critical Blind Spot in AI Search Agents: They Can't Handle Ambiguous Queries

Language models deployed as search agents have made impressive strides in retrieving information from the web. But a new research benchmark reveals a glaring weakness: these agents are essentially helpless when faced with ambiguous user queries that require asking clarifying questions to proceed.

The Problem: Ambiguity Left Unresolved#

Most existing benchmarks for evaluating search agents assume user requests arrive complete and unambiguous—an unrealistic premise that masks a real-world failure mode. When researchers at Stanford and elsewhere tested 17 different models on deliberately ambiguous search tasks, the results were stark. The strongest performer achieved just 13.73% accuracy, plummeting from 71.50% when given full context. Rather than struggling with reasoning, the agents revealed a more troubling pattern: systematic overconfidence in their answers despite incomplete information.

How InteractComp Works#

The researchers developed InteractComp, a benchmark containing 210 carefully curated questions spanning 9 domains. Each question was engineered using a "target-distractor methodology" to create controlled ambiguity that can only be resolved by asking follow-up questions. An example might be a query like "Find me papers on learning" that could refer to machine learning, educational research, or animal behavior—requiring the agent to disambiguate with the user.

Dramatic Recovery Through Forced Interaction#

The findings took an unexpected turn when researchers forced the agents to ask clarifying questions rather than plunging ahead. Under that constraint, performance skyrocketed, suggesting the latent capability was present all along—the agents simply weren't reaching for it. This points to a strategy or architectural failure rather than fundamental reasoning limitations.

A Growing Capability Gap#

Perhaps most concerning is a longitudinal trend: while general search performance improved roughly sevenfold over a 15-month period, interactive disambiguation capabilities stagnated entirely. This divergence reveals a blind spot in how the field has been optimizing language agents, with rapid gains in one dimension masking regression in another.

The benchmark and accompanying analysis provide both a diagnostic tool for identifying this failure mode and a training resource for addressing it, given the immediate feedback signals inherent to search tasks.