Researchers Identify Blind Spot in AI Agent Defense: Tool Hallucination That Existing Safeguards Can't Stop

LLM agents that rely on external tools suffer from a critical vulnerability that no existing security mechanism adequately addresses: they confidently call tools that don't exist in their registry and pass arguments that violate their declared schemas. Current defenses focus on selecting the right tool or constraining what agents can do once they've picked a real one—but they're blind to fabricated calls altogether.

The Structural Gap#

Tool-selection methods and security gates both share a fundamental assumption: the agent's function call refers to an actual, registered tool. This means hallucinated calls—by definition—fall outside any gate's scope. A gate cannot reject a call to a tool it doesn't know about, creating a structural blind spot that no existing defense can close. The researchers formalize this insight: hallucination defense must logically precede any security gating mechanism.

Mapping the Problem#

The study introduces a five-part taxonomy of tool hallucination (labeled H1–H5) and proposes the Resolution Rung, a training-free baseline resolver that checks whether a call references a registered tool and matches its signature. Across ten different hosted language models tested on two different invocation surfaces, the researchers measured 322 genuine hallucinations, with fabricated-tool calls heavily concentrated in unconstrained raw-JSON interfaces (34 instances versus 3 in structured surfaces). Notably, model size offered no protection—a 675-billion-parameter model showed hallucination rates comparable to models in the 7–8 billion range.

The Multiserver Challenge#

The problem intensifies when tools are served through the Model Context Protocol (MCP), where multiple servers merge into a single namespace. This creates collision and shadowing effects that expose new hallucination surfaces. Researchers identified a second taxonomy (M1–M5) to characterize these multiserver-specific vulnerabilities. Testing on the live MCP surface revealed 154 hallucinations, including cases where frontier models that performed cleanly in single-registry tests failed once multiple servers were combined.

Open Benchmarking#

To enable standardized evaluation of hallucination defenses, the team released the Hallucinated-Tools Benchmark (HTB), a versioned dataset allowing researchers to compare different resolver approaches across consistent test submissions.