Software release pipeline reported success despite skipping most quality checks

A production release pipeline's quality gate reported a PASS status on a build that failed to execute most of its assigned checks. The flaw allowed incomplete or skipped validations to score as perfect for two weeks before detection.

The Problem#

The blocking gate should have run comprehensive quality checks before releasing code to production. One subgate executed neither of its two required checks, while another ran only six of its eight. The gate's logic then drew its conclusion from a dataset that already excluded the unexecuted checks, treating missing data as a non-violation. The result: a nearly untested build received approval and rolled out successfully.

Nine Cases Converge on a Solution#

Researchers documented eight additional instances of the same problem pattern across different systems:

• Five emerged from the same production engagement
• Two appeared in open-source projects
• One cache gateway accepted a configured time-to-live parameter but never applied it during write operations
• An inference server credited available cache slots on every step, even for releases that returned no data
• One case was even introduced by the paper's author using tools designed to prevent exactly such failures

Though each case involved different systems and checks, they shared a common root cause and pointed to the same remedy.

The Fix: Add a Third State#

The solution introduces a third possible outcome beyond "pass" and "violate": unable to determine. This change converts silent passes—where missing checks generated false positives—into explicit failures and records the gap in exit status codes. The approach is straightforward: seven of the nine cases could be caught with a single query, command, or comparison.

Why This Matters#

The paper frames these nine cases not as novel failure types but as evidence that a single, falsifiable remedy can address diverse quality-gate blind spots across production systems. The fix emphasizes treating missing data explicitly rather than treating absence of evidence as evidence of absence in safety-critical gates.