28% of quantum transpiler bug fixes escape detection by standard equivalence testing

Quantum compilers rely on a standard verification method: checking that compiled circuits produce the same quantum operations (unitary transformations) as their source code. But this approach has a blind spot, and a new empirical study quantifies it for the first time.

Researchers conducted a systematic repository analysis of bug fixes merged into Qiskit, a leading quantum compiler framework, and found that nearly 28% of genuine bugs slip past equivalence-checking oracles — the standard tools used to validate compiler correctness. Even when augmented with additional checks for circuit validity, quality metrics, and performance improvements, these oracles still miss almost a third of real defects.

Why equivalence testing falls short#

Output-equivalence oracles operate under a narrow mandate: verify that a compiled circuit implements the same unitary transformation as the input, accounting only for global phase shift and qubit permutation. This design deliberately ignores other circuit properties — layout records, permutation metadata, phase information, and determinism in seeded runs — even when those records are publicly available.

The problem emerges at boundaries where circuit representations transform. When bugs corrupt layout tables or permutation records rather than the quantum operation itself, they remain invisible to standard verification. In the study's 68-fix core corpus, at least 15% represented this "unconditionally equivalence-invisible" class — defects that no enhancement to output checking could detect.

Scope and consistency#

The research team identified and dual-coded 68 Qiskit transpiler bug fixes using an independently validated taxonomy of fault manifestations. An extended corpus of 104 fixes maintained the same ratio at 27.9%, with tighter confidence intervals. The pattern is not unique to Qiskit: the same phenomenon replicates in tket (33% of 21 fixes) and shows up consistently, though at smaller scale, in Cirq.

Analysis of five low-cost pull-request characteristics revealed no systematic differentiators between invisible and visible bugs, suggesting the issue is structural rather than correlated with merge patterns or code review depth.

Implications for quantum software testing#

This gap concentrates at representation-boundary transitions — the architectural seams where quantum circuits move between internal formats. Because equivalence oracles by definition ignore non-semantic metadata, they cannot catch transformations that preserve the quantum computation while corrupting its administrative records.