AI
Datadog Adds Live Traffic Testing for AI Agents, Moving Beyond Offline Evaluation
The monitoring platform now lets teams run controlled experiments on agent variants against actual production requests, combining quality scores with operational metrics to validate real-world improvements.
InfoQ
Datadog has rolled out a new experimentation capability for its Agent Observability platform, letting teams test different versions of AI agents against live user traffic instead of relying solely on predetermined test datasets. The feature blends evaluation scores—measuring response quality and accuracy—with operational data like latency, token usage, cost, and error rates, so teams can verify whether modifications to models, prompts, retrieval methods, or available tools actually deliver measurable gains in production.
The Challenge of Lab vs. Real-World Performance#
An agent that scores well in a controlled test setting often behaves unexpectedly when confronted with unpredictable user queries. Offline evaluations remain valuable for catching known problems before deployment, but they can only be as thorough as the test datasets and scoring methods behind them. Edge cases, novel user workflows, and changing tool responses frequently expose blind spots that static test suites never anticipated.
How Online Experiments Work#
By running controlled experiments directly against production traffic, teams get a window into how an agent actually performs under realistic conditions. The system randomly distributes incoming requests between a baseline version and a modified variant, reducing the noise from differences in request patterns. Feature flags govern the traffic split, letting teams gradually roll out changes from a small user segment to a broader audience.
The approach reveals trade-offs that might be invisible in a single metric. A cheaper model might require more retries, or a new retrieval strategy could improve evidence quality while adding latency. Datadog layers its built-in evaluators—which score response quality, accuracy, and completeness—on top of conventional operational metrics, so teams see the full picture.
Tracing Down Unexpected Behavior#
When results diverge between variants, Datadog links experiment outcomes back to individual agent execution traces. Engineers can drill into tool calls, their parameters, and what the tools returned, exposing exactly why one version diverged from the other.
A Datadog case study compared GPT-5.6 Luna and GPT-6 Luna on an evenly split traffic test. An LLM-based evaluator assessed investigation quality on a one-to-five scale; both models scored nearly identically (3.84 vs. 3.86). The newer model delivered a 44.3% cost reduction—a significant win. But traces revealed it was generating invalid tool parameters more often, usually self-correcting through retries. The quality score alone would have masked this friction. Armed with that visibility, the team refined tool handling before greenlight.

