Withdrawn Study Flags How AI Agent Repair Rankings Shift Based on Evaluator Settings

A paper submitted to arXiv documents an instability problem in how agent-repair leaderboards rank competing methods. When evaluators are adjusted or reconfigured, the rankings shift—and a significant portion of this reordering stems from techniques that rely on evaluator signals while choosing among candidate repairs.

The Problem: Ranking Instability#

The core issue is that methods consulting evaluator-derived feedback during their internal repair-selection process can inadvertently create dependencies that cause leaderboards to reorder unpredictably. This coupling between evaluator configuration and method ranking undermines the reliability of public benchmarks meant to track progress.

The Dataset and Screening Approach#

The authors released AuditRepairBench, a paired-execution trace collection containing 576,000 registered cells (96,000 executed). The resource uses a modular screening framework with four interchangeable detection strategies: a learned influence proxy, a rule-based channel-exposure ratio that requires no trained model, a counterfactual sensitivity proxy, and a sparse human-audit proxy. These combine into a posterior that adjusts rankings and generates stratified leaderboard scores.

Validation and Results#

The team validated their approach on an 80-case subset through mechanism-anchored testing and an independent-discovery protocol where two separate annotator groups, blinded to the screening design, discovered coupling patterns and achieved pooled AUROC 0.83 on 79 cases. Their uncertainty propagation raised 95% coverage from 0.81 to 0.95.

When applied to reduce rank displacement, screening-guided blinding patches cut reordering by 55–74% (averaging 62%) with fewer than 50 lines of code. In contrast, random channel blinding achieved at most 7% reduction, and generic retraining produced at most 13% improvement.

AuditRepairBench-Lite, a rule-only variant tested on a 12,000-cell subset, preserved leaderboard order at Kendall τ = 0.88 in under 24 GPU-hours and serves as the primary release artifact at 42 GB.

Note#

The paper was withdrawn by author Yuelin Hu after initial submission on 6 May 2026 and a revision on 24 July 2026.