New Benchmark Tests LLM Coding Agents on Building Full Software Projects from Scratch

Coding agents built on large language models are increasingly capable of tackling larger development challenges beyond isolated code fixes. Yet measuring their ability to construct entire software projects remains difficult. A new benchmark aims to close that gap.

What is E2E-SWE?#

Researchers have created E2E-SWE, a standardized evaluation framework designed to assess whether AI coding agents can build complete, functional software repositories from the ground up. The benchmark comprises 186 full-repository generation tasks spread across 11 different programming languages. Each task challenges an agent with nothing but a natural-language specification and an empty workspace—requiring it to deliver a finished, installable project that passes a hidden test suite.

Building Tasks with Rigor#

The construction process for each benchmark task emphasizes precision and implementability. Software engineers work alongside LLMs to craft detailed, language-agnostic specifications and corresponding test suites. To validate that tasks are well-formed and genuinely solvable, the researchers then put them through an iterative auditing stage. During this phase, autonomous agents themselves inspect tasks for defects, using both static code analysis and results from live model runs to identify and repair problems.

Performance Across Frontier Models#

Testing 13 state-of-the-art models revealed substantial gaps in capability. Pass rates at the first attempt (pass@1) ranged from 11.7% to 67.7%, demonstrating both clear model differentiation and substantial room for improvement. Analysis of how agents solve these tasks uncovered long, deliberate reasoning chains concentrated early in the problem-solving process, underscoring the importance of planning and system-level thinking required to assemble working codebases from scratch.

Why It Matters#

Moving beyond isolated code changes to evaluating repository-scale generation represents a meaningful step toward practical AI software engineering. By establishing standardized, specification-independent tasks, E2E-SWE provides a common yardstick for tracking progress in this critical capability.