DevRev launches benchmark that actually tests enterprise AI where it matters

Vendors flood the market with claims that their AI agents are ready for the real world, but determining which ones truly deliver remains murky. DevRev, an enterprise platform backed by Nutanix co-founder Dheeraj Pandey, contends that existing benchmarks miss the mark because they emphasize abstract problem-solving over the sprawling, messy realities of corporate data.

This week, the company introduced its Enterprise AI Agent Benchmark, an openly accessible test suite that publishes its dataset, evaluation tools, scoring methodology, results, and execution traces. The inaugural release covers the two foundational tiers of a four-level autonomy framework, with an invitation for the community to develop the upper levels.

Why enterprise benchmarks miss the point#

Current standards like TAU-Bench and Agent's Last Exam reward exotic reasoning capabilities, but enterprise work rarely demands that kind of sophistication. Instead, the real challenge lies elsewhere. "We are not going to get closer to benchmarking the needs of enterprise by creating more complex tasks," says Ahmed Bashir, DevRev's chief technology officer. The breakthrough insight came from recognizing that moderate-difficulty work becomes time-consuming not because it's intellectually hard, but because relevant information is scattered across incompatible systems, formatted inconsistently, and locked behind varying permission levels.

Bashir frames this as a distinction between task complexity—largely a solved problem—and data complexity. Enterprise data exists in silos that persist despite decades of integration efforts, organized unevenly, and protected by fragmented access controls. That's where difficulty actually lives.

The architecture of a realistic test#

To translate this insight into a measurable framework, DevRev created what it calls scale-invariant ground truth. A single reference dataset representing a mid-size software company exists at four scales: 1x, 4x, 16x, and 64x. The correct answer to any task stays the same across all versions; the additional data serves as noise that a well-functioning agent should ignore. At the smallest scale, roughly 40 percent of available records relate to a typical question. At the largest, that drops to just 2.5 percent.

The design exposes a critical weakness in many AI agents: they excel at finding relevant needles in small haystacks, but that strategy collapses when volume hits production levels. The benchmark evaluates performance across three dimensions—precision (correct answer with traceable source), (cost proportional to question complexity, not data size), and (permission boundaries respected, all actions auditable).