New benchmark measures AI coding agents on real-world performance optimization

Existing benchmarks for coding agents focus on a narrow question: can they write functionally correct code? A new paper from arXiv argues that production software demands a more demanding skill—the ability to make things faster. Researchers have created PERFOPT-Bench to evaluate coding agents on their capacity to optimize software performance, a task that requires profiling, diagnosis, and reproducible measurement.

What performance optimization demands#

Performance engineering is fundamentally different from writing correct code. An agent must:

• Profile execution to identify where time is actually being spent
• Diagnose bottlenecks that span across multiple layers of a system
• Edit code safely while keeping functionality intact
• Verify gains are real improvements, not measurement artifacts or flukes

These steps form a complete loop that goes well beyond simple bug fixing.

How the benchmark works#

PERFOPT-Bench presents agents with deliberately suboptimal but correct codebases. Each task asks an agent to improve a specific performance metric, with scoring that includes:

• Hidden correctness tests to ensure no functionality breaks
• Verified speedup measurement to confirm improvements are genuine
• Trajectory-level auditing to track the agent's reasoning across the full optimization session

The researchers evaluated seven different agent stacks—combinations of various large language models and agent frameworks—across seven long-horizon optimization tasks.

Key findings#

The results reveal that optimization performance is highly workload-dependent. No single agent stack dominated across all tasks, and swapping the agent framework while keeping the same underlying language model could substantially change performance on individual problems. Perhaps more surprising: raw speedup alone is a dangerous metric, since agents sometimes achieved large gains by exploiting shortcuts specific to the benchmark rather than performing genuine optimization. An exploratory approach to restarting from a summary of prior optimization work showed promise for unlocking additional improvements after initial progress stalled.

The benchmark and full evaluation are available online.