New Benchmark Reveals LLM Agents Struggle with Evidence-Based Diagnosis in Cloud Outages

Researchers have released Cloud-OpsBench, an evaluation framework designed to test how well large language model agents can diagnose problems in cloud-native systems and justify their findings. The work addresses a critical gap: existing benchmarks measure whether agents arrive at the right answer, but don't assess the quality of their investigative process.

The Problem with Existing Approaches#

Current methods for evaluating AI diagnostic tools fall short in two ways. Static benchmarks provide consistent test cases but offer limited interaction with real system tools, while live testing environments expose realistic diagnostic interfaces but make it hard to compare results fairly because system states differ between runs. Most importantly, both approaches focus almost entirely on final answers rather than how agents actually build their case.

What Cloud-OpsBench Offers#

The new benchmark comprises 754 verified test cases spanning 57 different failure modes across two microservice applications: OnlineBoutique and TrainTicket. Each failure is captured as a system state snapshot and then replayed through standard diagnostic tools and APIs. Crucially, the framework provides not just correct answers but also diagnostic evidence graphs—the breadcrumb trails an agent should follow to reach its conclusion. This enables researchers to compare not just whether agents got it right, but how they got there.

Key Findings: Answer Correctness Masks Deeper Weaknesses#

When tested against ten different LLM agents, results revealed a troubling gap between correctness and reasoning. The strongest agents achieved root cause accuracy rates of 76% on OnlineBoutique and 68% on TrainTicket. However, their Evidence Closure Rates—a measure of how thoroughly they documented the investigation path—reached only 38% and 15% respectively. This outcome-process mismatch demonstrates that final-answer accuracy substantially overstates how well these agents can perform evidence-grounded diagnosis.

Why This Matters#

For cloud operations teams relying on AI to troubleshoot incidents, a correct diagnosis is only half the battle. Operators need to understand why a system failed so they can prevent recurrence and validate the fix. An agent that guesses right without solid supporting evidence is far less trustworthy than one that methodically traces the problem and shows its work.