Large language models struggle with quantum program repair across software versions

Researchers testing large language models' ability to repair quantum programs have uncovered a fundamental problem with how such benchmarks are validated: fixes that work in one version of a library may silently fail in another, making benchmark scores unreliable across different software releases.

The Challenge of Version-Dependent Bugs#

The study evaluated how well four leading LLMs—GPT-4o-mini, GPT-5o-mini, GPT-5.4, and GPT-5.4-mini—could fix real quantum defects using Bugs4Q, a standard benchmark of 67 genuine Qiskit programming errors. The researchers re-tested all entries against six different pinned releases of Qiskit (ranging from 0.25.0 to 2.3.1), discovering that many benchmark labels became invalid without raising any errors.

This phenomenon, termed silent label inversion, occurs when either a reference fix stops executing properly or the buggy program ceases to reproduce the original failure—both situations obscuring the true state of the code. As a result, a repair's correctness cannot be judged by the benchmark alone, but depends critically on which version of the library is being used.

Performance Across Models and Versions#

When tested with up to 10 repair attempts per defect, GPT-5.4 emerged as the best performer with a pass@10 rate of 48.8%, followed closely by GPT-5.4-mini at 47.3%. The smaller models lagged significantly: GPT-5o-mini achieved 30.3%, while GPT-4o-mini managed just 22.6%. Performance peaked on Qiskit 0.45.0 but declined sharply after the Qiskit 1.0 transition.

Root Causes Behind Failures#

Rather than pointing to flawed repair logic, many failures traced back to deprecated or incompatible application programming interfaces introduced between versions. Notably, 64% of repairs that succeeded did so only on entries that were actually invalid under the target version being tested—a finding that underscores the unreliability of version-agnostic evaluation.

Implications for Future Benchmarking#

The research underscores that benchmark validation must precede repair evaluation. The authors have released a re-validated, version-pinned version of Bugs4Q to address these shortcomings, establishing clearer ground truth for future assessments of LLM-based code repair in the quantum domain.