Researchers Develop Method to Spot Unreliable Tests in Cyber-Physical Systems

Testing cyber-physical systems—software that controls physical hardware in everything from thermostats to autonomous vehicles to medical devices—presents a distinctive challenge: these systems often operate in unpredictable environments where timing, synchronization issues, and unexpected events can cause identical tests to produce different results. When a test passes in one scenario but fails in another despite no code changes, engineers call it a "flaky" test.

The Problem with Flaky Tests#

Flaky test cases undermine the reliability of testing results. Engineers may spend significant effort investigating what they believe are genuine bugs, only to discover the problem stems from test instability rather than actual software defects. This wastes resources and reduces confidence in the entire testing process. The conditions responsible for these inconsistent outcomes—what researchers term "flaky conditions"—can be difficult to pinpoint, especially in complex systems operating under uncertain circumstances.

A New Integrated Approach#

Researchers have developed a unified testing framework called HyTestTF that builds on earlier work in hybrid model-based testing. The approach combines three integrated techniques:

• HyTest: A test generation and execution method for hybrid systems
• TReVa: A validation technique that confirms the reliability of testing results
• FlaRe: A new method that identifies flaky test cases and the specific conditions causing them

By performing these checks during early-stage development with just one additional round of testing, the framework enables teams to quickly spot and differentiate between genuinely flaky tests and reliable ones.

Why It Matters#

For safety-critical systems like medical devices and autonomous vehicles, understanding which tests are truly reliable is paramount. Reliably filtering out flaky tests means engineers can focus on actual defects rather than chasing false positives caused by environmental variability or timing issues.