Why Modern AI Systems Need Microsecond-Speed Caching, Not Milliseconds

Building reliable, fast systems often means stepping back to rethink fundamental requirements rather than patching problems as they arise. Dumanshu Goyal, a lead engineer at Airbnb who oversees the company's data platform, frames this principle through an analogy: NASA's Space Shuttle program spent decades (and billions of dollars) maintaining 24,000 fragile silica heat tiles, only to eventually switch to SpaceX's simpler capsule design that required far less infrastructure. The lesson applies to modern data systems too.

In a talk on open-source Valkey architecture patterns, Goyal argues that today's AI-driven applications are creating new performance demands that challenge the millisecond-latency assumptions built into many caching systems over the past 15 years.

The AI Feature Store Bottleneck#

The pressure comes from AI inference pipelines that need to assemble predictions in strict time budgets. A concrete example: DoorDash's fraud detection and recommendation system must complete predictions within 100 milliseconds. Sounds generous until you examine what happens inside.

A single prediction request requires hundreds of features—pieces of real-time data like whether a credit card is new, recent login counts, order history, and dozens more. To meet the deadline, the system fans out these requests in parallel. But parallel requests introduce a classic problem: the slowest call determines overall latency. In a relay race with hundreds of runners, one straggler sets the pace.

This is where tail latency (the p99 percentile) becomes critical. While average response times might hover around 1 millisecond, occasional spikes to 10 milliseconds or worse are inevitable at scale. Since these feature requests eat into the 100-millisecond budget, and the AI model itself already consumes low single-digit milliseconds, the underlying feature store must deliver sub-millisecond response times—often in the hundreds of microseconds range—to leave the AI model enough room to compute.

From Redis to Valkey: A Shifting Landscape#

Redis, which emerged in 2009, became the default caching layer for building real-time systems. Its simplicity, advanced data structures, and broad ecosystem made it pervasive across infrastructure. For over a decade, millisecond-latency architectures were considered fast enough.

In 2024, however, a licensing dispute prompted the Linux Foundation to back a fork called Valkey, which maintains full compatibility with Redis while remaining fully open-source. Within its first year, Valkey gained community traction comparable to PostgreSQL in developer surveys—a testament to the appetite for an open alternative.