New Benchmark Reveals AI Agents Struggle to Implement Features from Documentation

Researchers have introduced a novel benchmark for testing whether AI-powered software engineering agents can translate documentation updates into functional code implementations. The evaluation reveals significant gaps in current agent capabilities, with even the strongest systems succeeding on less than 40% of tasks.

The Benchmark and Its Scope#

Doc2Feat-Bench contains 634 tasks spanning 10 mature open-source projects, encompassing roughly 114,000 lines of code changes in total. Rather than relying on GitHub issue descriptions—which often include implementation-level hints—the benchmark uses actual documentation changes as the source of truth. Each task pairs a documentation modification with its corresponding code implementation and developer-written tests, creating a more realistic evaluation scenario.

The dataset covers a diverse range of software categories: libraries, developer tools, and frameworks. This breadth aims to reflect real-world development conditions and the variety of challenges engineers encounter across different project types.

Evaluation and Results#

Researchers assessed multiple state-of-the-art agents using two common scaffolding approaches. The highest-performing system achieved only a 37.72% success rate (OpenHands paired with Qwen3-Coder-480B), despite consuming substantial computational resources measured in token usage.

To enable more focused evaluation with limited resources, the team also curated Doc2Feat-Bench Verified, a subset of 114 manually reviewed instances where task clarity and evaluation validity have been confirmed by human reviewers. This curated version provides a more reliable foundation for benchmarking while reducing computational overhead.

Key Challenges Identified#

Analysis of agent failures points to three critical bottlenecks:

• Cross-file editing: Agents struggle to coordinate changes across multiple files in a cohesive manner
• Module comprehension: Understanding existing code structures and dependencies remains difficult
• Semantic interpretation: Accurately parsing what documentation changes actually require proves problematic

These findings suggest that current approaches to agentic software development may be overestimating agent readiness for practical, end-to-end feature implementation tasks.