Agentic AI
New framework pinpoints where AI coding agents fail in complex repositories
TrajAudit uses intelligent diagnosis to cut through noise in lengthy AI agent execution traces, helping developers understand why automated coding systems stumble.
Researchers have developed TrajAudit, a diagnostic system designed to automatically identify failure points in AI agents that handle large-scale coding tasks. The framework tackles a growing challenge: as software agents take on more ambitious repository-level work—from bug fixes to new feature implementation—pinpointing exactly where and why they go wrong becomes critical for both refinement and reliability.
The core problem#
Traditional failure diagnosis methods struggle with coding agents because of two stubborn obstacles. These systems leave behind lengthy execution trails that can span dozens or hundreds of steps, making it hard for large language models to trace the causal chain of a failure. Additionally, those trails overflow with noise—redundant code snippets, verbose context, and low-value observations—that can confuse and mislead the reasoning process.
How TrajAudit works#
The framework deploys an investigator agent equipped with two key capabilities. First, a semantic saliency folding module strips away noise by removing context irrelevant to the failure. Second, a diagnostic guidance module extracts clues from test failure reports as prior knowledge, helping the agent focus investigation on the most likely problem areas. Rather than forcing the agent to process the entire messy trajectory at once, it can selectively inspect folded sections on demand—preserving access to full context without the cognitive overload.
Benchmarking the approach#
The researchers introduced RootSE, a benchmark containing 102 real-world instances from repository-level coding tasks, each with annotations marking the earliest decisive error step and supporting justification. Testing on this benchmark showed TrajAudit outperforming strongest competing approaches by 10.8% and 21.6% in exact failure localization accuracy, depending on whether reference information was available.
The work opens new possibilities for making agentic coding systems more transparent and easier to improve iteratively.
