Claude leads in agent-building benchmark but clears only 24% of tasks

Researchers have created a benchmark to measure how well AI systems can autonomously design and construct other AI agents—a task that remains surprisingly difficult. The leading performer, Claude Opus 5 running in Claude Code, achieved just 23.9% on the test, revealing significant gaps in how machine builders approach engineering challenges.

The Hyper-τ-bench benchmark#

Sierra, an enterprise AI company founded by tech veteran Bret Taylor, open-sourced Hyper-τ-bench in September as a follow-up to its earlier τ-bench framework. Where the original benchmark measured how well a finished agent could interact with users and follow policies, this new version flips the problem: it asks whether a developer agent can build that customer-service system entirely on its own.

The test presents a developer agent with business materials—documents, transcripts, APIs, and codebases—and tasks it with constructing a customer service agent under strict model and cost constraints. The finished agent is then evaluated against simulated conversations across airlines, retail, telecom, and banking sectors. The agents pass when they provide accurate information and execute the correct changes in backend systems.

Sierra tested six configurations pairing major AI models with coding environments, including Claude in Claude Code, OpenAI's models in Codex, and Moonshot AI's Kimi K3. The results leaderboard shows Claude Opus 5 narrowly ahead of GPT-5.6 Sol at 22%, with none of the autonomous setups exceeding 25%.

Why low scores matter#

The single-digit and low double-digit performance rates don't automatically signal a broken benchmark. Tests designed for frontier AI systems should be challenging enough to expose real differences and leave room for improvement. But what stands out in these findings is the 82.2% achieved by human-AI pairs—a substantial gap that comes with important caveats.

Sierra's researchers describe this as an "oracle reference" rather than typical human performance. The human-AI combination worked with frontier models but crucially had access to ground-truth requirements that the autonomous agents had to discover independently. In other words, the comparison isn't quite apples-to-apples, though the gap still points to genuine capability differences.

Where autonomous agents fail#

The developers dug into what went wrong during agent construction and found recurring patterns that echo familiar engineering problems: