New Framework Validates LLM-Generated API Tests Against Real Execution

Testing REST APIs at scale requires orchestrating sequences of API calls that respect dependencies between operations and their outputs. Language models have emerged as a promising way to infer these dependencies from API specifications, but a critical gap remains: most approaches treat LLM-generated relationships as fact without ever confirming them through actual execution.

The Validation Problem#

Current LLM-based API testing systems can introduce spurious dependencies that don't actually exist, miss valid operation chains, and generate test sequences that fail when executed. This mismatch between theorized and practical dependencies wastes testing effort and leaves bugs undetected.

How APIPilot Works#

Researchers propose APIPilot, a framework that grounds LLM reasoning in execution feedback. The system operates in three stages:

• Hypothesis generation: Using OpenAPI specifications, APIPilot applies structural patterns and LLM-based semantic analysis to identify candidate producer-consumer relationships between API operations.
• Validation through execution: Rather than assuming these dependencies are correct, the framework tests each one in the real API before incorporating it into the test model.
• Refinement from responses: As tests run, runtime responses update the model: resource pools are refreshed, input constraints adjusted, and invalid mappings pruned or corrected.

The validated dependencies feed into a dependency graph, from which APIPilot constructs test workflows using bounded graph traversal that balances coverage with feasibility.

Real-World Results#

Testing across 16 production REST APIs, APIPilot achieved:

• 92.3% operation coverage: The system exercises nearly all available API endpoints
• Up to 58.6% code coverage: Test sequences reach a majority of backend logic
• 88.1% workflow execution success: Generated tests actually complete without errors

The framework outperformed both pure LLM-based approaches and traditional rule-based testing baselines. Beyond coverage metrics, APIPilot detected 197 unique server failures and exposed discrepancies between published API specifications and their actual behavior—revealing practical value in the execution-first philosophy.