Study reveals widespread configuration problems in AI agent setup files

A new study from arXiv has catalogued the first systematic inventory of misconfiguration patterns in files used to instruct coding agents, revealing just how often software teams make the same preventable mistakes.

The Problem#

As coding agents have become more common tools for automating software engineering work, developers rely increasingly on configuration files—typically named AGENTS.md or CLAUDE.md—to spell out rules about code architecture, development workflows, naming conventions, and testing standards. Yet little research has examined what goes wrong in these files or how often projects stumble on the same pitfalls.

What the researchers found#

Scientists conducted a grey literature review and analyzed 100 widely-used open-source repositories that contained either an AGENTS.md or CLAUDE.md file. They uncovered six distinct configuration smells and built automated detection heuristics for each. The prevalence was striking:

• Lint Leakage (62% of files) — the most widespread problem
• Context Bloat (42% of files)
• Skill Leakage (35% of files)
• Conflicting Instructions, Incomplete Documentation, and Missing Scope

The research also discovered that certain smells frequently cluster together; Context Bloat, Skill Leakage, and Conflicting Instructions often appear in the same configurations.

Why it matters#

As organizations scale up their use of AI agents to handle routine coding tasks, the configuration files that guide these tools become critical infrastructure. Widespread configuration smells suggest that many teams lack best practices or tooling to validate their agent setups, potentially leading to agents that misbehave, exceed their intended scope, or contradict one another. The automated detection heuristics described in the paper could help developers catch these issues before deployment.