LLMs and AI agents increasingly rely on tests to guide software development decisions

Tests now play a central role in how large language models and software engineering agents make decisions about code. Rather than simply verifying a finished product, tests increasingly guide program construction, direct repair work, filter candidate solutions, constrain transformations, and provide execution data for analysis. This shift builds on test-driven development principles while departing from them in fundamental ways.

How testing shapes AI-driven coding#

The role of tests in LLM-based development diverges from traditional practices in several key dimensions. Test timing, whether test oracles are available upfront, which code artifacts can be edited, and how execution results factor in—all differ significantly from conventional test-driven workflows. These variations affect which decisions tests actually influence and how effectively they guide AI agents.

A structured landscape of test-driven practices#

A new scoping survey mapping this terrain identifies five distinct testing patterns:

• Red-Green-Refactor cycle: the classical approach, where a failing test drives implementation
• Test-conditioned generation: tests inform code generation before execution
• Execution-guided refinement: running tests shapes iterative code improvement
• Test-mediated analysis: tests enable deeper program understanding and localization
• Evaluation-only testing: tests measure outcomes without directing the generation process

The review synthesizes evidence from 87 research records across nine major software engineering tasks: code generation, repair, translation, refactoring, clone detection, code search, fault localization, training-data construction, and formal-specification validation.

What the evidence reveals#

A critical finding cuts across all applications: passing tests alone does not guarantee behavioral equivalence, meaningful feedback, or process correctness. Test availability, test validity, feedback quality, and evaluation independence should be treated as separate properties. Aggregate improvements across experiments can mask divergent results for particular models, specific tasks, or different measurement approaches.