New Benchmark Tests AI's Ability to Diagnose Why Long-Horizon Agent Tasks Fail

When an AI agent executing a complex, multi-step task fails, understanding why it failed requires more than just noting the unsuccessful outcome. Developers need to trace through the execution history to identify which component or role was responsible and locate the earliest critical error that set the failure in motion. A new benchmark from arXiv researchers addresses this challenge for long, intricate agent trajectories.

The Challenge of Long-Trajectory Failure Diagnosis#

Existing benchmarks for failure attribution focus primarily on shorter execution traces, leaving a gap in the field's ability to evaluate diagnosis across hundreds or even thousands of recorded steps. The median failure trajectory examined in this work spans 145 steps—substantially longer than what prior tools handle well. At this scale, manual inspection becomes impractical, yet automated systems struggle to distinguish between downstream symptoms and upstream root causes.

Introducing LongRCA Bench#

The researchers present LongRCA Bench, a dataset of 1,140 naturally occurring failed trajectories across five different domains. Importantly, these failures were not artificially injected; they represent genuine execution breakdowns. Each trajectory is independently annotated by humans with two key labels: the responsible role (which component or agent bears responsibility) and the earliest decisive root-cause step (the point where the decisive error originated). These dual annotations recognize that pinpointing responsibility and localizing root causes are distinct diagnostic challenges.

Baseline Performance and RCTA Method#

Current best-performing baselines reach only 13.2% accuracy when asked to identify the exact root-cause step. The researchers propose Root-Cause Trajectory Attribution (RCTA), a training-free approach that works by retrieving candidate error steps from segment summaries and tracing them backward to earlier handoff instructions. Using the same evaluation framework as the baselines, RCTA achieves substantially better results: 51.1% accuracy for responsible-role attribution and 24.1% accuracy for exact root-step localization.

Why This Matters#

These results underscore a crucial finding: responsible-role attribution and exact root-step localization should be evaluated as separate diagnostic objectives in long-trajectory failure analysis. The performance gap between current methods and the benchmark difficulty signals a genuine need for more sophisticated approaches to failure diagnosis in realistic, long-horizon agent systems.