Automatically Generated Failure Taxonomies Help AI Agents Learn From Their Mistakes

Improving an AI agent system often requires understanding why it fails. Techniques like trajectory selection, prompt optimization, and runtime monitoring can all enhance performance without retraining the underlying model—but they rely on analyzing raw execution traces that tend to be verbose, instance-specific, and inconsistent in their language. A new research paper proposes a more structured alternative.

The Problem With Raw Traces#

Current feedback mechanisms operate on unprocessed execution logs, which present several challenges. These traces are lengthy and filled with details irrelevant to broader failure patterns, making it hard to spot recurring issues. They also lack a consistent vocabulary: the same underlying failure might be described differently across runs, hindering the accumulation of actionable insights. This inconsistency makes it difficult for improvement procedures to learn from past failures in a systematic way.

How AdaMAST Works#

Researchers have introduced AdaMAST, a system that automatically constructs a compact, evidence-grounded failure taxonomy directly from an agent's behavior. Rather than relying on hand-written failure codes or human annotation, the method extracts failure names, definitions, and detection patterns entirely from the traces themselves. The resulting taxonomy organizes failures along three dimensions: system-level issues, role-specific problems, and domain-specific challenges.

The compressed vocabulary is orders of magnitude smaller than raw traces while still preserving meaningful distinctions between different failure modes. Importantly, the taxonomy is adaptive—one induced for a particular domain shares few codes with taxonomies derived from other domains, reflecting domain-specific failure patterns.

Demonstrated Improvements#

Testing shows the approach delivers concrete benefits across multiple improvement mechanisms:

• Agent system search: Taxonomy-coded diagnoses of failed candidates outperformed open-ended reflection across all five benchmarks tested
• Runtime feedback: When integrated with SWE-agent, the taxonomy increased resolution on SWE-bench Verified Mini from 60% to 70%; it also boosted Claude Code's performance from 64.0% to 70.7% as a runtime skill
• Trajectory selection: AdaMAST-Judge, a verifier built on the induced codes, improved best-of-5 accuracy on Terminal-Bench 2.0 by 8–15 percentage points compared to Pass@1 baseline