Agentic AI
New Benchmark Tests AI Coding Agents on Real-World Optimization at Repository Scale
Researchers introduce FormulaCode, a dataset of nearly 1,000 performance bottlenecks from scientific Python projects to measure how well AI agents can improve large codebases simultaneously for correctness and speed.
Recent advances in large language model (LLM) coding agents have pushed them beyond fixing isolated bugs toward tackling optimization challenges across entire repositories. Yet assessing their true capabilities remains difficult, as most existing benchmarks rely on artificial test cases, simple pass-fail grading, or single-outcome measures that don't reflect the real-world complexity of improving large software systems.
A New Benchmark for Realistic Optimization#
Researchers have unveiled FormulaCode, a comprehensive evaluation dataset designed to measure how well LLM agents optimize authentic codebases. The benchmark draws on 957 real performance bottlenecks identified in scientific Python repositories hosted on GitHub. Each bottleneck comes with expert-vetted fixes and, on average, roughly 265 community-generated performance tests, creating a demanding evaluation landscape that mirrors actual development constraints.
Why This Matters for AI Coding#
Existing code-focused benchmarks often fall short in critical ways. They typically evaluate narrow tasks in isolation, apply only binary right-or-wrong scoring, or focus on a single objective such as speed or correctness. FormulaCode moves beyond these limitations by requiring agents to juggle multiple competing goals—maintaining functional correctness while simultaneously improving performance. This holistic test better captures what real engineers face when tuning production code.
What the Results Show#
Early evaluations of frontier LLM agents against FormulaCode reveal significant gaps. Optimizing entire repositories while respecting realistic correctness and performance constraints remains a substantial hurdle for even the most advanced models. The findings suggest that repository-scale, multi-objective code optimization is still an open challenge in AI-assisted software engineering.
The project's full materials, including the benchmark itself and evaluation details, are available through the researchers' project website.
