AI
Popular AI Coding Benchmarks Show Reliability Issues, Study Finds
A systematic audit reveals that leading performance-optimization benchmarks for evaluating coding agents suffer from runtime instability and inconsistent scoring rules that can obscure true progress.
Researchers conducting an analysis of three major coding-agent benchmarks—GSO, SWE-Perf, and SWE-fficiency—have uncovered significant reliability problems that undermine confidence in their leaderboard rankings. The study, published on arXiv, reveals that these widely-cited evaluation frameworks conflate technical instability with genuine algorithmic progress.
The Stability Problem#
The team replayed official reference patches for 740 code-optimization tasks across different machine configurations to test benchmark consistency. They found that most reference patches could be re-run successfully, but critical problems emerged when checking whether they maintained validity across hardware setups. Only 39 of 102 GSO tasks, 11 of 140 SWE-Perf tasks, and 411 of 498 SWE-fficiency tasks consistently satisfied the original benchmark rules on every cross-machine replay. SWE-Perf proved especially fragile, with many reference patches delivering negligible runtime improvements—making it difficult to distinguish real optimization gains from measurement noise.
Scoring System Inconsistencies#
The researchers also examined how benchmark scoring rules affect rankings. Among eight public submissions tested on both GSO and SWE-fficiency, the official leaderboard scores disagreed on 9 out of 28 pairwise comparisons between submissions. More troublingly, SWE-fficiency's scoring mechanism dramatically overweights its ten most difficult tasks, assigning them score contributions between 58.5% and 82.8%—distorting the overall picture of agent performance.
What Gets Lost in Translation#
Across all three benchmarks, the researchers found that at least one public submission matched or exceeded the reference patch on 85.3% of replay-valid tasks. This suggests that aggregate leaderboard scores may be hiding important variations in per-task difficulty and success rates. The audit identifies which tasks produce reliable performance signals and quantifies exactly how much individual problems contribute to final rankings.
