Agentic AI
New framework reveals hidden failures in AI-powered business analytics agents
Researchers present a methodology to evaluate enterprise analytics agents across semantic understanding, execution quality, and reliability—revealing significant gaps that final-answer grading alone would miss.
Enterprise analytics agents face a measurement crisis. These systems do far more than convert natural language to database queries—they must grasp business context, pick the right data sources, choose metric definitions, run multiple tools, and deliver findings that shape real operational and financial decisions. Yet conventional evaluation methods grade only the final answer, leaving critical failures invisible.
Researchers have now proposed a comprehensive end-to-end, trace-backed methodology to expose where these agents actually falter. Instead of asking "did we get the right number?" the framework asks whether the agent understood the question, executed correctly, and behaved consistently.
How the Methodology Works#
The approach evaluates agents across three core families:
- Semantic understanding: Does the agent grasp what the user is really asking?
- Execution quality: Did it pick the right tools, sources, and definitions to answer?
- Reliability: Does it produce consistent results across repeated runs?
Each evaluation run undergoes validity checks and receives tiered scores that feed into a decision framework rather than a simple pass/fail gate. The methodology draws on question banks with human-authored gold-standard answers, multiple repeated runs per question, and detailed runtime traces that expose the agent's reasoning step-by-step.
Real-World Test Results#
Researchers tested the framework on an internal analytics agent used by a large online marketplace, running 300 traces across 50 real business questions, two model configurations, and three randomized repetitions each.
The findings were sobering. A higher-capability model configuration showed meaningful gains—it nearly eliminated early refusals (dropping from 73% to 0%) and more than tripled the rate of real-data answers (21% to 73%). Yet the improvements came with hidden costs:
- Tool exhaustion: The agent ran out of allowed tool calls on 16% of attempts
- Schema exploration overruns: 77% of traces exceeded budget exploring the database structure
- Consistency failures: Across 50 questions, the agent reinterpreted table meanings 41 times
- Decomposition gaps: Both configurations struggled to break down finance questions into their required components, even when source table selection and escalation improved
