DeltaSelect cuts AI coding agent testing costs by 58% while maintaining accuracy

Evaluating coding agents is expensive. Researchers at arXiv have released DeltaSelect, an open-source approach designed to make rapid iteration on AI coding systems more affordable while preserving the quality of performance signals.

The Problem with Full Benchmarks#

Comprehensive coding-agent benchmarks exist to rank models, but they're poorly suited for the constant comparison cycles that happen during development. Running a complete suite is time-consuming and costly; results vary between runs; and the testing harness in practice often differs from the one used in published benchmarks. A resampling analysis of DeepSWE's published results revealed a significant gap: only 19.5% of tasks (22 of 113) showed a Pearson correlation of 0.50 or higher between single-run performance and full-benchmark results. In other words, most individual tasks fail to reliably predict overall progress.

How DeltaSelect Works#

The method operates in three steps. First, it identifies tasks whose one-time results consistently align with full-benchmark performance using Pearson correlation as the statistical measure. Second, it converts fractional verifier scores into a unified metric using linear regression. Third, it selects a fixed set of tasks that fits within a specified dollar budget—allowing teams to pick the most informative subset for their constraints.

DeltaSelect is built for repeated baseline-versus-candidate comparisons during active development, not for final model rankings. The distinction matters: developers need fast feedback loops, while external benchmarks need comprehensive coverage.

Real-World Case Study Results#

Researchers applied DeltaSelect to refine custom skills and instructions in a gpt-5.6-luna low-reasoning setup, running 13 evaluations. The cost per evaluation dropped to USD 27.86 total at August 2026 rates. The final adopted version was 58.1% cheaper than the starting point (USD 1.75 versus USD 4.18; p=0.008), and the calibrated score improved despite the cost reduction (42.36% versus 36.46%), with no statistically significant variance difference (p=0.326).

This combination—lower expense, higher measured performance, equivalent reliability—suggests the method successfully filters out noise-prone tasks while preserving signal.