Agentic AI
Search-Based Framework Dramatically Improves AI's Ability to Diagnose Why Agents Fail
A new iterative method helps language models find root causes in long agent execution logs by conducting wider searches for diagnostic clues rather than settling on the first plausible explanation.
As AI agents take on increasingly complex, multi-step tasks, the execution logs they generate during failures have become enormous and difficult to analyze. Pinpointing what went wrong—the root cause—is essential for improving reliability, but manually reviewing these massive traces is impractical. Current automated approaches using large language models (LLMs) fall short, particularly when handling long sequences of actions, because the key pieces of evidence tend to be scattered, sparse, and far removed from the moment of failure itself.
The Problem with One-Shot Diagnosis#
Existing root-cause attribution (RCA) methods typically ask an LLM to examine an execution trace once and render a diagnosis. This approach works reasonably well for shorter logs but breaks down at scale. The model tends to lock onto the first explanation that sounds plausible and moves on, never digging through the rest of the trace to find stronger evidence. Searching through a massive log for the true culprit is thus framed as a fundamentally different challenge than a simple judgment call.
Continual Search: An Iterative Approach#
Researchers introduced Continual Search, a framework that nudges the LLM judge repeatedly over multiple turns to keep hunting for unresolved diagnostic clues. Rather than stopping at the first diagnosis, the system encourages the model to continue examining the trace and reassess its conclusions in light of new findings—much like a human investigator might revisit evidence after spotting anomalies.
Benchmarking at Scale#
To test the method rigorously, the team created MegaRCA-Mix, a new benchmark that addresses a glaring gap in existing RCA datasets: the lack of large-scale, long-horizon execution traces. This dataset comprises 50 human-annotated failure cases drawn from complex, execution-heavy tasks—providing a genuinely challenging testbed.
Results across multiple benchmarks were striking. On MegaRCA-Mix, Continual Search boosted Opus-4.8's F1 score by 29 percentage points, from 0.471 to 0.608. A counterintuitive finding emerged as well: within the same model family, smaller models equipped with the iterative search method often outperformed larger models using one-shot diagnosis. This suggests that smart search strategy can compensate for raw model capacity.
