Runtime verification moves to agents themselves—but the question of what they test against remains critical

AI coding agents are shifting verification into their own execution loops, moving beyond static code analysis toward runtime testing. Tools like Greptile, Cursor, and Devin now execute the code they generate and read back the results before any human review occurs. As Stripe's internal agents demonstrate—shipping over 1,000 reviewed pull requests weekly—this feedback cycle prevents bottlenecks from overwhelming senior engineers.

However, a critical gap remains: the gap between testing an isolated service against mocks and testing against the full, interconnected system where the code will actually run.

The consensus on runtime verification#

The trend across agent tools is unmistakable. Static analysis has hit its ceiling. Reading a diff and running unit tests against mock data cannot catch race conditions that emerge under real requests, regressions that only surface after rendering, or integration failures that hide between services.

Greptile's TREX feature now executes changes in sandboxed environments and returns logs and traces. Cursor's cloud agents clone repositories into virtual machines for build-and-test cycles. OpenAI's Codex Cloud and Devin's full environments with shells and test runners all follow the same principle: give agents a runtime where they can prove their work before handoff. This shift is necessary and happening rapidly across the industry.

The hidden limitation of current approaches#

The approach all these tools share has a built-in boundary. They equip agents roughly with what sits on a developer's desk: the immediate service, locally-startable dependencies, and mocks standing in for everything else. For a single, self-contained application, this suffices.

For cloud-native systems composed of many interdependent services, this covers less than half the ground that matters. The expensive bugs live at service boundaries:

• Contract drift between services where API assumptions diverge
• Serialization mismatches at integration points
• Cascading failures in retry or timeout policies that break two hops downstream
• Non-functional regressions like performance degradation, resource contention, and security issues that only surface under realistic load and traffic

None of these appear in unit tests or against mocks. They surface only when code runs against real services, real data, and real system dynamics. A sandbox environment, by design, cannot reach these layers.