AI
Study reveals how AI agents fail silently when APIs withhold data
Researchers find that LLM-based agents struggle to distinguish empty results from genuine API errors, often inventing answers instead.
An LLM agent interacting with a production API faces a blind spot: when a query finds no matching data, the server returns the same success response (HTTP 200 with a valid body) as when it simply fails to understand the request. This ambiguity leaves agents unable to detect silent failures—and the results can be troubling.
Researchers from arXiv (cs.SE) audited over 700,000 API parameters across 2,500 OpenAPI specifications to understand how often silent failures occur and what happens when AI agents encounter them.
The Schema Problem#
The core issue lies in how API specifications are written. The audit found stark differences in how constraints are documented:
- Only 7.5% of APIs declare enumerated values (fixed lists of valid inputs)
- A mere 15.2% encode any machine-readable constraint at all
- Yet 40.1% of documents describe constraints in plain English prose that their machine-readable schemas omit entirely
This gap creates a trap for AI agents, which rely on formal schema definitions to validate inputs.
Machine-Readable vs. Prose-Only Constraints#
The team tested this hypothesis by running 219 perturbations against 27 commercial API vendors through a single aggregation layer (Monid). The results were striking:
- Machine-checkable constraints (e.g., enums): honest error responses in 111 of 111 tests
- Prose-only constraints: silent failures in 44 of 61 cases (statistically significant, p = 2e-13)
Constraint format, not vendor identity, predicted whether an API would actually return an error or fail silently.
How Agents Respond to Silent Failure#
Twelve LLM models from eight different families were then run against these endpoints on real tasks. When agents encountered vocabulary that descriptions merely exemplified but didn't formally specify, they failed in every single case (88 of 88 attempts). However, vocabularies listed completely in the schema worked correctly 88–91% of the time.
When running the full agent loop and encountering silent failures, the models exhibited concerning behavior:
- Detected the failure in only 12% of cases
