Agentic AI
New Testing Method Finds Bugs AI Developers Miss by Starting with Product Specs
Researchers propose a novel approach to testing code written by AI agents, deriving test properties from requirements documents instead of the implementation itself.
As AI agents increasingly take on the task of writing software directly from specifications, the code arrives faster than human reviewers can inspect it—and tends to fail in predictable ways. A team of researchers has developed a new testing method that exploits the gap between what code is meant to do and what it actually does.
The Problem with Current Testing#
Property-based testing (PBT) is a mature technique that checks whether a program's core invariants hold across many random inputs. Recent work has pushed LLM agents to write these properties themselves, cutting the labour cost of property discovery. But existing implementations work backward: they read the finished code and derive properties from it. This means the properties themselves inherit any biases baked into the implementation—missing bugs that contradict the original intent.
A Specification-First Approach#
The researchers propose Spec-Driven PBT, which inverts this logic. Rather than mining properties directly from the implementation, their agent first reads the specification (such as a product requirements document) and explicitly writes down what the code should guarantee in plain language. Only then does it inspect the actual code to translate those requirements into runnable tests.
The intuition is straightforward: a specification records the intended behaviour, while implementation details can quietly violate it. By anchoring tests to documented intent, the method avoids being misled by poor choices elsewhere in the code.
Results: Complementary Coverage#
The team evaluated Spec-Driven PBT against the traditional code-driven approach across 25 applications from ViBench. The findings reveal:
- Different bugs caught: Nearly 60% of the 169 bugs found across both methods were unique to one approach. Spec-Driven PBT and Code-Driven PBT discovered largely non-overlapping sets of failures.
- Fewer tests, better targeting: Spec-Driven PBT generated 2.4x fewer property tests while surfacing just as many bugs overall, suggesting properties grounded in requirements are more efficient at uncovering flaws.
- Distinct failure patterns: Bugs unique to the spec-driven method tended to violate documented workflows and feature contracts. Code-driven bugs concentrated on robustness issues and edge cases within the implementation itself.

