Repository Context Files Don't Boost Coding Agent Performance, Study Finds

Developers routinely create context files like AGENTS.md to orient coding agents toward their repositories' specific practices and conventions—a strategy that's widely endorsed by AI model providers. Yet a new study questions whether this common practice actually delivers results.

The Research Question#

Researchers examined whether repository-level context files genuinely help coding agents complete real-world tasks more effectively. To answer this, they tested agents across two distinct scenarios: established benchmark tasks from well-known repositories paired with AI-generated context documentation, and a new set of actual issues pulled from repositories where developers had already written context files.

Key Findings#

The results were striking. Adding context files did not meaningfully increase task completion rates across the board. More surprisingly, including these files boosted inference costs by over 20% on average—a real penalty with no corresponding performance gain.

The pattern persisted regardless of which language model powered the agent or which agent framework was used. Both automatically generated and human-authored context files showed the same lack of benefit.

The researchers did identify one exception: instructions within context files were generally followed reliably by coding agents. However, broader repository overviews—components that model providers actively recommend—proved unhelpful for improving task outcomes.

Implications for Practice#

The work suggests that while context files have value for documenting non-standard coding practices, teams should verify their effectiveness through rigorous testing rather than assuming they'll enhance agent performance. The research underscores the importance of empirical validation before deploying optimization strategies in production workflows.