Flaky quantum tests are rare but hard to catch—study of Qiskit Terra reveals hidden reliability risks

Researchers have completed the first comprehensive study of flaky tests—those that pass and fail unpredictably without any code changes—in quantum software, exposing a subtle but important reliability challenge.

The team examined Qiskit Terra, IBM's widely-used quantum computing library, by running its full test suite 10,000 times across 23 releases in controlled laboratory settings. The work reveals that while flakiness is uncommon in the tested environment, when it does occur it tends to stick around—and requires far more test runs to detect than standard development practices typically allow.

What the data showed#

Across nearly 27,000 test cases, the researchers identified just 62 tests that exhibited flaky behavior. Yet the pattern of recurrence tells a more interesting story: 83.87% of flaky tests reappeared across multiple releases, suggesting that once a test becomes unreliable, it tends to remain problematic. Only 16 tests were isolated to a single release.

The failure probabilities varied wildly, with half the flaky tests failing at rates at or below one in a thousand executions. This statistical spread creates a thorny detection problem: discovering these failures with confidence would require thousands to tens of thousands of test runs—far beyond what most continuous integration systems budget for routine builds.

Why quantum testing is different#

Quantum software presents unique challenges absent in classical computing. Tests that exercise quantum hardware or simulators can be sensitive to timing, environmental noise, and subtle interactions in ways that traditional software testing often avoids. The study's controlled laboratory protocol—which standardized execution conditions across releases—may partly explain the relatively low observed flakiness rates compared to what teams might encounter in real-world CI/CD pipelines with varying system loads and configurations.

What happens next#

The researchers have released their full dataset of per-test execution outcomes to support future studies and replication efforts, laying groundwork for developing detection techniques suited to quantum software's specific demands.