AI
Measurement Methods, Not Quality, May Determine Whether AI Code Lasts
New research shows that how researchers measure code survival—not the code itself—drives conflicting conclusions about whether AI-generated code persists longer in repositories than human-written code.
When researchers study how long code from AI agents survives after being merged into software projects, they reach dramatically different conclusions. Some find that agent code lasts longer than human code; others say it lasts about as long; still others claim it gets modified sooner. A new study suggests the disagreement may stem not from the code quality itself, but from how the researchers chose to measure survival.
The Core Finding#
Researchers analyzed over 854,000 lines of code across nearly 2,800 merged pull requests in the AIDev dataset, then recalculated survival metrics using 88 different combinations of measurement choices. For code produced by supervised AI agents like GitHub Copilot, the methodological decisions altered the size of effects but kept their direction consistent. However, for code from the fully autonomous agent Devin, the measurement choices actually flipped the conclusion entirely.
How Methodology Changes the Verdict#
The turning point came down to statistical clustering. When researchers treated individual code lines as independent observations, 15 of 22 specifications showed that Devin's code was modified sooner than human code. But when they adjusted standard errors by clustering results at the repository level, only 3 specifications reported that difference—no better than what random permutation of labels would produce. Even within single repositories, no measurement strategy detected any meaningful difference.
Classification and Intent Disagreement#
The researchers also tested two approaches for identifying why code gets modified. Large language model classifiers agreed with human labels about as reliably as two humans agree with each other (kappa = 0.69–0.72), whereas keyword-based rules performed poorly (kappa = 0.38). Critically, these two methods reached opposite conclusions about whether modifications to agent code stem from necessity or quality issues, demonstrating that the tool chosen for analysis can reshape interpretation.
Broader Implications#
When the team applied five previously published study designs to their data, each one changed its result depending on a single methodological choice. The researchers emphasize that human developers modify agent code most frequently under virtually every measurement scenario, yet the narrative about whether agent code "survives" ultimately depends as much on how the question is framed as on the code itself.

