How Elastic Built a Shared Evaluation Framework for Production AI Agents

Building multiple AI agents across an organization quickly leads to a common problem: each team reinvents evaluation practices from scratch. Susan Chang, principal data scientist at Elastic, described how her company moved from isolated, ad-hoc evaluation efforts to a shared framework that lets teams rapidly spin up comprehensive testing—without duplicating work.

The Multi-Agent Challenge at Elastic#

Elastic, the company behind Elasticsearch and Kibana, has deployed AI agents across diverse use cases: security threat detection (Attack Discovery), enterprise chatbots built on proprietary data in Elasticsearch, and observability-focused agents. Each serves a fundamentally different purpose—yet each team was separately designing datasets, custom metrics, tracing pipelines, and evaluation logic.

For the security team, agents needed to identify cyber-attacks in petabyte-scale log repositories. Success meant measuring precision and recall on alert matching, factuality of findings, and correctness of MITRE framework classifications. Meanwhile, the chatbot team faced an entirely different problem: answering user questions about Elastic products using retrieval-augmented generation (RAG). Their evaluation criteria included response relevance, completeness, and correct generation of ES|QL query syntax. Neither team's tools or metrics translated to the other's domain.

Tracing as the Foundation#

Before evaluation comes visibility. Chang emphasized that comprehensive tracing is non-negotiable—without granular data about what an agent does internally (which tools it calls, what databases it queries, how it reasons), surface-level metrics like a final answer being wrong become actionable diagnosis. Elastic explored multiple tools including LangSmith and Phoenix, eventually using a combination to capture every step of agent execution.

A feature like LangSmith's "Add to Dataset" proved particularly valuable: when a user reports an agent failure via feedback, that real-world scenario can be instantly recorded and added to the test suite for future regression checks. This transforms production incidents into permanent, reusable test cases.

From Silos to Shared Infrastructure#

Rather than accept redundant evaluation stacks, Elastic's teams (with support from data scientists working across Python and TypeScript codebases) consolidated toward a single, modular framework. The shared toolkit handles: