AI
New Benchmark Tests Whether AI Coding Agents Can Build Full Repositories From Scratch
Zero2Repo challenges coding agents to construct complete software projects from requirements alone, across multiple languages—and current leading systems struggle on real-world tasks.
Researchers have introduced Zero2Repo, a benchmark that measures whether coding agents can move beyond patch-fixing to actually construct working software repositories from the ground up. The challenge reflects a gap in how AI systems are currently evaluated: while they're increasingly deployed to build new code, most existing benchmarks focus on single languages and rely on artificially constructed tests.
How the Benchmark Works#
The Zero2Repo evaluation presents agents with three inputs: a product requirements document that specifies what the software should do, an interface contract defining key APIs, and an empty workspace. The agent must then produce a complete, functioning repository that would run in that project's native ecosystem.
The tasks themselves are derived from real open-source projects through a language-agnostic pipeline. The process converts actual, version-pinned codebases into formal behavioral specifications, reproducible build environments, and hidden acceptance tests. Crucially, each task is validated upfront: a reference implementation extracted from the original project must pass the tests, and adversarial checks confirm that incorrect code fails them.
Rigorous Evaluation Without Shortcuts#
The testing setup is deliberately stringent. Agents run in isolated containers with acceptance tests withheld until an explicit submission is made. Scoring is binary—either all tests pass or the submission fails—with no LLM judge applying subjective partial credit. This mirrors how real software deployment works.
The current benchmark covers Python, TypeScript, Go, and C++, demonstrating that the pipeline makes no language-specific assumptions and can extend across mainstream ecosystems.
Early Results Show Room for Improvement#
Results reveal a sobering picture for frontier models: on 11 tasks drawn from repositories likely present in training data, the strongest agent solved only 10. Every failed submission cleared 90–99% of the hidden tests, suggesting agents are close but miss critical details.
Notably, the researchers found that 67–100% of failures in the two strongest performers traced to a single omitted feature or a low-frequency rule mentioned in the specification—not a missing entire subsystem. This suggests the gaps are concrete and addressable rather than fundamental architectural shortcomings.
