New Benchmark Reveals Language Model Agents Still Fall Short at Production Incident Investigation

Researchers have released ORCA-bench, a new evaluation framework designed to measure how well large language model agents can perform oncall work—the critical task of diagnosing system failures in production environments. The results paint a sobering picture: even the most capable models achieve only modest accuracy rates when asked to investigate incidents using real observability tools.

The Challenge of Production Diagnostics#

Root cause analysis in live systems requires far more than code comprehension. Engineers must synthesize information from multiple sources—noisy metrics, logs, distributed traces, and source code—to trace back from a user-facing symptom to its underlying cause, sometimes hours after an incident began. This demands not just pattern matching but genuine reasoning about how systems interact and fail.

ORCA-bench pairs over 1,000 incident investigation tasks with six days of continuous telemetry collected from a microservice architecture running under simulated user load. The benchmark grants agents access to real observability interfaces: Prometheus for metrics, Jaeger for distributed traces, and OpenSearch via Grafana for log aggregation. Agents can also examine the application's full source code.

Accuracy Gaps Across Difficulty Levels#

Testing five frontier models, researchers found stark performance gaps. On Medium-difficulty tasks—representing realistic incident reports—the best-performing agent achieved 25.3% accuracy. Hard-difficulty cases showed even sharper decline, with accuracy dropping to 10.0%. Importantly, these results held even when testing Claude Fable 5, one of the latest frontier models.

Weaker models exhibited concerning failure modes: hallucinating implausible root causes in roughly 40% of incident reports. Removing source-code access degraded performance further for every model tested, both reducing overall accuracy and increasing spurious diagnoses.

A Realistic but Limited Test#

The testbed itself—a curated 50 GB snapshot spanning six days—represents a deliberately controlled environment. Tasks systematically vary report specificity, time-to-detection, and overlapping fault scenarios, with ground-truth symptoms validated and signed off by expert site reliability engineers. Human scorers independently re-evaluated the LLM-as-judge assessments, achieving strong agreement (Cohen's κ_w = 0.91).