New benchmark reveals why code agents struggle with real-world feature implementation

A new evaluation framework exposes a critical gap in how repository-level code agents approach real-world programming tasks. While these AI systems have shown promise at feature addition—the process of extending existing codebases with new functionality—existing tests measure only whether the final code works, offering little insight into how agents actually reason through the problem.

Introducing RACE-bench#

Researchers have developed RACE-bench, a reasoning-augmented benchmark designed to evaluate code agents on feature addition tasks in production repositories. The dataset comprises 528 real-world feature requests sourced from 12 open-source projects, each paired with executable patch verification and detailed intermediate reasoning steps that mirror how developers actually approach the work.

Rather than treating agents as black boxes, RACE-bench tracks their decision-making across four key stages: understanding the issue description, identifying which files need modification, planning concrete implementation work, and decomposing the solution into manageable steps.

The evaluation reveals a reasoning divide#

Testing three representative repository-level code agents on the full benchmark showed resolved rates ranging from 29% to 70%—a wide spread indicating substantial differences in capability. But the real insight emerged when researchers examined how agents reasoned through tasks.

High-level problem comprehension proved to be agents' relative strength. However, the translation from understanding the requirement to writing actual code showed significant degradation. Patches that technically applied to the codebase but failed test validation revealed a troubling pattern: they covered 35.7% fewer of the reference reasoning steps that human developers would consider and introduced 94.1% more unsupported reasoning than successful patches.

Why this matters#

The findings underscore that measuring patch correctness alone masks deeper problems in agent reasoning. A patch that passes superficial checks may have skipped crucial reasoning steps or made unfounded assumptions. This distinction becomes critical for real-world adoption, where code quality depends not just on whether tests pass, but on whether the solution reflects sound engineering judgment.

The research framework opens a path toward improving repository-level code agents by making their reasoning processes visible and measurable.