Momentic Scales Caching to 20 Billion Entries by Migrating from PostgreSQL to ClickHouse

Momentic, a company building AI-powered software testing tools, successfully redesigned its caching layer by moving from PostgreSQL to ClickHouse, enabling it to serve over 2 million queries daily across 20 billion cache entries with typical response times around 250 ms.

The Scaling Problem#

As Momentic's testing platform grew, its cache table exploded from roughly 80,000 to 1 billion entries, revealing PostgreSQL's limitations under strain. The concurrent read-write pressure created resource contention and locking issues that worsened as data volume climbed by multiple orders of magnitude. The team needed a database architecture better suited to high-velocity, high-cardinality workloads.

Why ClickHouse Fit the Use Case#

ClickHouse's column-oriented structure and sparse primary indexing proved substantially different from PostgreSQL's B-tree approach. Rather than scanning progressively larger result sets as data grows, ClickHouse uses "granules" to narrow searches when key values are known—a property that keeps query costs roughly flat regardless of total database size.

By designing a composite primary key combining test ID, step ID, platform version, branch name, and commit timestamp, Momentic achieved efficient lookups in 90% of cases, particularly for feature-branch queries.

Handling Edge Cases at Scale#

The migration revealed a challenge: main-branch queries occasionally needed to scan nearly all data parts, causing memory spikes and disk thrashing. Momentic resolved this by adding materialized views that precompute available timestamps per test ID, allowing the system to jump directly to relevant data sections.

The team also simplified its query pattern, replacing three separate PostgreSQL queries with a cleaner single-operation flow using ClickHouse's ReplacingMergeTree: insert new cache entries, re-insert accessed items to refresh their time-to-live, and let ClickHouse handle deduplication asynchronously. This efficiency gain was substantial enough to eliminate an otherwise necessary Redis caching layer entirely.

Safe Cutover Strategy#

Rather than a risky big-bang switch, Momentic ran both systems in parallel, mirroring writes to PostgreSQL and ClickHouse while serving all production traffic from PostgreSQL. ClickHouse ran in "shadow" mode, with results validated against PostgreSQL before gradually shifting live traffic over. Even post-cutover, dual writes continued briefly to preserve a quick rollback path if needed.