How to actually evaluate whether coding agents work

The argument that coding agents cannot be rigorously evaluated is seductive on its surface. Software engineering sprawls; requirements are fuzzy; codebases harbor years of undocumented choices; two developers can solve the same problem in completely different valid ways. Agents are also non-deterministic—they may succeed one run and fail the next. Yet this reasoning conflates "hard to measure" with "unmeasurable." Traditional software systems face these same challenges, and we evaluate them routinely. Coding agents should follow the same path.

The agent is the whole system, not just the model#

When we speak of evaluating a coding agent, we're not testing a language model in isolation. An agent comprises the underlying model plus its harness, tools, repository context, prompts, permissions, and execution environment. Tweak any single component—swap in better tooling, tighten feedback loops, improve context retrieval—and performance shifts materially. A larger model paired with weak repo context may underperform a smaller one with sharp tools and fast tests.

This detail matters for benchmarks. A published score typically reflects a specific model-plus-environment combination operating under fixed token and time budgets—not the model's innate ability. Conflating the two leads to misinterpretation and wasted comparisons.

Grade on behavior, not diffs#

One common objection holds that a code-writing agent might produce a correct patch that looks nothing like a human-authored reference version, making exact-match grading impossible. That's true. But it's not a blocker; it's a redirect.

Instead of comparing diffs, start the agent at a known repository state, assign it a concrete task with available context, and evaluate the resulting codebase against observable contracts:

• Does it compile or build?
• Do existing tests remain green?
• Do newly added hidden tests for the requested behavior pass?
• Are public APIs and data formats unchanged?
• Do schema migrations run bidirectionally?
• Are performance and resource ceilings respected?
• Did the agent stay within the allowed scope?
• Do linters, type checkers, or security scanners flag new issues?

This approach allows many different implementations while keeping the success criteria objective and hard to fake. Some properties—maintainability, naming, architectural alignment—still require human judgment, just as they do in code review.