Study reveals stark quality gaps among five autonomous coding agents—some beat humans, others fall short

Researchers analyzing nearly 38,000 pull requests submitted by autonomous coding agents have found striking differences in code quality and maintainability—differences that vary as much by which agent created the code as by how that code compares to human work.

The study, which examined contributions from five commercial agents (OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code) across 2,807 GitHub repositories between December 2024 and July 2025, upends the assumption that agent-generated code performs uniformly. Instead, it points to a fragmented landscape where some agents substantially outperform human developers while others struggle.

The Performance Spread#

Results vary sharply by vendor. OpenAI Codex-authored pull requests were reverted roughly half as frequently as human-submitted code (6.1% reversion rate versus 11.5%), suggesting more durable and acceptable contributions. In contrast, Devin code faced reversion rates above 14%—notably higher than the human baseline. GitHub Copilot, Cursor, and Claude Code fell somewhere between these extremes.

Post-merge code churn—the volume of changes and fixes required after a PR lands—also diverged by agent, pointing to differences in how well each system anticipates reviewer feedback and real-world maintenance demands.

Security Gains, but Nuanced#

Agent code pooled across all vendors showed a measurable security advantage: it was less likely than human code to contain security smells like hardcoded credentials and unsafe eval-style constructs. The protective effect was moderate but consistent (odds ratio 0.63). However, this aggregate improvement masks vendor-specific variations, suggesting that security benefits concentrate in some agents more than others.

Uneven Review Burden#

Beyond code metrics, the research uncovered disparities in how human reviewers engage with different agents. Copilot PRs drew the most human review attention and change requests, implying that code flagged as needing more scrutiny. Claude Code contributions experienced the longest wait for first review—a median of 12.6 hours—potentially signaling either lower priority or reviewer uncertainty.

The researchers released their pipeline code, statistical reports, and datasets to support replication and further investigation.