OpenAI Uncovers 18-Year-Old libunwind Race Condition Using Population-Level Crash Analysis

OpenAI engineers tracking down mysterious crashes in Rockset—the C++ data service powering ChatGPT's search and data plugins—discovered that what appeared to be a single intractable bug was actually two separate problems colliding simultaneously. The breakthrough came not from diving deeper into individual failures, but by shifting to what they call epidemiological debugging: analyzing the full population of crashes across a year of production to spot patterns rather than reasoning about isolated incidents.

The Debugging Breakthrough#

Instead of focusing on individual crash reports, the team created an automated pipeline to process every core dump from the past twelve months. ChatGPT itself helped write scripts that extracted register state, filtered noise, and classified failures into categories like return-to-null, misaligned-stack, and others. Running this analysis across the entire Rockset dataset revealed two distinct crash populations with completely different characteristics—a finding that would have remained hidden if examining cases one by one.

The misaligned-stack failures were geographically concentrated in a single Azure region, appeared on a specific date, and never affected long-running instances. This pattern pointed to a single faulty host where the CPU was silently producing incorrect arithmetic without raising exceptions. Removing that host from service eliminated the misaligned-stack crashes entirely.

Isolating the Real Culprit#

With hardware-related failures separated out, the remaining return-to-null crashes became solvable. The team had previously ruled out C++ exception unwinding because they thought they had counterexamples—crashes in code paths that didn't use exceptions. But those counterexamples belonged to the hardware-corruption cluster. Once that contamination was removed, every remaining crash was occurring during exception unwinding, revealing a race condition in GNU libunwind.

The 18-Year-Old Race Condition#

The vulnerability exists in libunwind's _Ux86_64_setcontext function, used during C++ exception handling. The function rebuilds a ucontext_t struct on the stack with the desired register state, then switches control to a cleanup handler. The bug: the stack pointer (%rsp) is updated to point to the new stack frame before the instruction pointer is read from the old struct. During that window—just one instruction wide, roughly 100 picoseconds—the struct becomes unprotected by the kernel's red zone guarantee. If a signal arrives in this microsecond gap, the kernel writes its signal frame over the struct, corrupting the instruction pointer and causing a jump to null or garbage.