Adrian Cockcroft: Why AI-Powered Performance Tools Must Move Beyond Simple Percentiles

Adrian Cockcroft has spent four decades hunting down performance problems in systems at Sun Microsystems, Netflix, eBay, and Amazon—starting with deep dives into kernel source code and evolving into approaches powered by generative AI. In a recent conversation with RedMonk analyst Rachel Stephens at P99 CONF, he outlined why traditional performance metrics fall short and how machine learning is reshaping how engineers diagnose latency issues.

The Evolution: From Kernel Archaeology to Vibe Coding#

Cockcroft's journey reflects the broader shift in how performance engineering gets done. Early in his career at Sun, understanding system behavior meant reading kernel source code line by line to grasp what utilities like vmstat were actually measuring. The insight gained from that meticulous work fed into influential books on Solaris tuning and resource management.

Today, the toolkit has vastly improved—end-to-end tracing platforms now offer rich visibility. Yet Cockcroft still looks for what existing tools miss: the anomalies hiding in plain sight. When everything appears normal in standard dashboards but the system behaves poorly, he digs deeper, shifts his analytical lens, or moves from aggregate numbers to distributions. Using what he calls vibe coding—building custom tooling with AI assistance in minutes rather than hours—he no longer faces the friction of mastering new languages or hunting down libraries.

"My speedup is infinite, because this code would never exist without these tools," he noted. "I wouldn't have the time to build them."

The Limits of Percentiles: Why P99 Can Mislead#

Cockcroft's central critique: percentile-based metrics like P99 obscure the true shape of latency in real systems. A single number cannot reveal whether response times cluster around one mode or multiple. In a common scenario—cache hits fast, cache misses slow—the distribution often has two distinct peaks. As cache hit rates shift, each peak stays at the same latency value, yet both the average and P99 swing wildly, falsely suggesting the system's behavior changed when really only traffic composition shifted.

"Percentiles don't work when trying to understand the latency and performance of modern web services," he said.

The fix: move from reducing distributions to single figures, to tracking the peaks themselves. Instead of collapsing data into an average, a better approach identifies multiple modes and watches how their prominence fluctuates. Cockcroft built open-source tooling in R—leveraging ChatGPT to handle a language he hadn't used in years—to automate this analysis. The tool treats response-time histograms as distributions worth exploring rather than numbers to be averaged away.