Audit Reveals Praxa AI Pipeline Metrics Measure Wrong Things Despite Looking Correct

Researchers conducting a measurement audit of the Praxa AI pipeline have documented a critical problem in how agent evaluations are interpreted: metrics can appear numerically sound while actually measuring something entirely different from what their labels suggest.

The Core Problem#

The study examined evaluation reports, implementation records, and operational data from Praxa AI systems and discovered pervasive gaps between what metrics claim to measure and reality. A routing report logged 112 passes against 27 failures—suggesting high reliability—yet zero gating failures occurred because known gaps were explicitly exempted from the evaluation criteria. This hollow pass rate illustrates how measurement definitions can render reported numbers meaningless.

Latency and Timing Discrepancies#

The audit found dramatic timing misrepresentations. An export containing 8,843 tool attempts included 8,395 recorded durations and 448 missing values. Among the recorded durations, 121 entries equaled 2,147,483,647 milliseconds—the maximum value for a signed 32-bit integer—indicating database clamping of abandoned client connections.

When comparing recorded versus server-observed latencies, the pooled 99th percentile appeared to be 2,147,483,647 ms in the dataset, versus just 38,118.31 ms in actual server observations. This wasn't a treatment effect but a stratification artifact, underscoring how data collection boundaries can distort performance pictures.

The Compaction Pilot Case#

A documented optimization pilot reported a 94.39% reduction in follow-up inputs. However, when examining the complete flow combining trigger and follow-up calls together, the actual reduction dropped to 46.54%. The partial view created an inflated headline finding.

Verification and Transparency#

The researchers reproduced all descriptive calculations, verified 91 timing statistics through independent weighted rational-arithmetic implementation, and executed 13 scoring-function and 12 analysis-verifier tests. They published ancillary code and data to support reproducibility, including SQL queries, Python analysis scripts, and raw telemetry files.

Finite-completion bounds show how missing durations constrain any statements about aggregate timing without requiring value imputation—a methodological contribution for handling incomplete operational data.