New Benchmark Reveals LLMs Struggle With End-to-End Software Generation

Researchers have developed a fresh approach to measuring how well large language models can generate working software from the ground up—and the results suggest this remains a significant challenge for even the most capable systems.

The Problem with Existing Benchmarks#

Current software generation benchmarks have a fundamental flaw: they provide predefined templates and structural scaffolds that reduce the task to simply filling in missing code sections. Meanwhile, they rely on rigid unit tests that verify internal implementation details rather than validating whether the final product actually works as intended from a user perspective. This means existing benchmarks don't truly capture what it takes to build a complete, usable application from nothing.

Introducing CLI-Tool-Bench#

To address these gaps, researchers unveiled CLI-Tool-Bench, a new benchmark purpose-built for evaluating real 0-to-1 software generation. The evaluation framework tests agents' ability to autonomously design and construct command-line interface tools without relying on predefined structural templates.

The benchmark comprises 94 high-quality, real-world repositories spanning multiple programming languages and varying degrees of complexity. Crucially, agents begin with an entirely empty workspace—they must independently manage repository planning, handle dependencies, and make architectural decisions. Generated software is then executed in isolated sandboxes, with its output and system-level behavior compared to human-written reference implementations using a rigorous multi-tiered equivalence metric that prioritizes black-box testing over white-box unit tests.

What the Results Reveal#

Testing seven state-of-the-art LLMs exposed sobering limitations:

• Success ceiling remains low: The highest-performing models achieved only a 43.8% overall success rate
• Monolithic code patterns: Agents show a consistent tendency to generate single-file, monolithic structures rather than modular designs
• Token consumption paradox: Using more tokens during generation does not translate to better task performance

These findings underscore that building complete software systems from scratch represents a frontier challenge that current LLMs have not yet mastered, despite their impressive capabilities in narrower software engineering tasks.