Automated Production Benchmarks Bridge Gap Between Lab and Real-World AI Coding Agents

Evaluating AI coding agents in production is fraught with trade-offs: A/B testing takes weeks and risks user experience, shadow deployments produce unreliable signals across runs, and standard public benchmarks misalign with real-world code patterns and developer workflows.

Researchers have developed REAP (Relevance and Execution-Audited Pipeline), an automated system that transforms actual developer-agent sessions into production-grounded benchmarks without requiring manual curation. The approach tackles a fundamental challenge facing organizations deploying coding AI at scale: how to gather trustworthy evaluation signals quickly and at the speed of production infrastructure changes.

The Core Challenge#

Public benchmarks and laboratory conditions often diverge from production reality in three critical ways:

• Language distribution mismatch between test datasets and deployed systems
• Prompt style differences between synthetic test scenarios and real developer requests
• Codebase structure gaps that make lab results poor predictors of field performance

Large monorepos compound the problem further: their build infrastructure is unstable, and tasks become stale as the codebase evolves. This means any benchmark must be continuously updated to remain valid—a requirement that makes manual quality control impractical.

How REAP Works#

Rather than relying on humans to validate tasks, REAP layers automated verification into the pipeline:

• LLM-based task classification to filter out prompts that cannot be meaningfully tested
• Agentic test-relevance validation to ensure tests actually verify the agent's solution
• Multi-run stability checks to identify and exclude flaky tests that produce inconsistent results

These safeguards ensure that the resulting benchmarks yield reproducible, trustworthy signals without the friction of manual auditing.

The Harvest Benchmark#

The team applied REAP to create Harvest, a benchmark where each task consists of a real developer prompt paired with fail-to-pass tests recovered from production. The dataset spans four programming languages, with the majority of tasks drawn from Hack.