New Benchmark Reveals How Far AI Falls Short at Optimizing Real-World Code Performance

A new research benchmark exposes a significant gap between state-of-the-art language models and expert software engineers when it comes to optimizing code performance in real repositories. The gap is large: AI agents manage to achieve less than a quarter of the speedup that human experts accomplish on the same tasks.

What is SWE-fficiency?#

Researchers have introduced SWE-fficiency, a benchmark designed to measure how well AI systems can perform performance optimization work in production-grade codebases. The benchmark contains 498 distinct optimization tasks drawn from nine major open-source projects used in data science, machine learning, and high-performance computing—including numpy, pandas, and scipy.

Each task presents the same challenge an optimization engineer faces: given a complete codebase and a workload that runs slowly, the agent must examine the code, identify performance bottlenecks, write a patch to fix them, and verify the patch doesn't break existing tests.

How the benchmark was built#

Rather than inventing synthetic optimization problems, the researchers mined real-world solutions from GitHub. Their automated pipeline scraped performance-improving pull requests, using keyword filtering, static analysis, and code coverage tools to validate that proposed fixes actually deliver the claimed speedups and don't introduce test failures. This approach grounds the benchmark in genuine engineering work rather than hypothetical scenarios.

Where AI stumbles#

When researchers evaluated current leading-edge AI agents, the results were sobering. Across the board, the models struggled with three core challenges:

• Pinpointing bottlenecks: Agents have difficulty narrowing down which parts of a codebase actually cause slowness
• Reasoning across functions: Understanding how performance issues propagate through interconnected code remains difficult
• Maintaining correctness: Proposed optimizations often break tests or alter program behavior unintentionally

Why this matters#

Most existing software engineering benchmarks measure whether an AI system can identify what needs to be fixed. SWE-fficiency shifts the focus to the harder problem: to fix it. Performance engineering demands understanding code semantics, respecting correctness constraints, and reasoning about execution flow—skills that current language models have barely begun to master at scale. The researchers are releasing both the benchmark and their data collection pipeline to support future work on automated performance optimization and long-horizon code reasoning.