Safety-Aware Mutation Testing Proposed for Autonomous Driving Systems

A team of researchers has proposed a new approach to testing autonomous driving systems that addresses a fundamental gap in current safety validation methods. The work, presented as a vision paper on arXiv, introduces Safety-Aware Mutation Testing (SAMT)—a shift away from traditional testing methods toward a more systematic evaluation of interaction failures between system components.

The Problem with Current Testing#

While simulation-based testing is crucial for autonomous driving safety, the field lacks a clear mechanism for determining when sufficient test coverage has been achieved. Existing metrics typically evaluate individual components in isolation or treat the entire system as a black box, missing critical component interactions that contribute to most real-world autonomous driving accidents.

Traditional mutation testing—which deliberately introduces faults into software to measure test quality—hasn't been effectively adapted for autonomous driving systems. Simply applying code-level mutations to individual ADS modules fails to capture the complex interactions that actually cause failures.

A Systems-Thinking Approach#

SAMT takes a different approach by systematically injecting temporally bounded faults into the messages exchanged between ADS modules. This simulates realistic interaction failures rather than individual component faults.

To ensure these artificial faults represent genuine hazards, the researchers propose deriving mutant generation rules directly from established safety engineering frameworks, particularly System-Theoretic Process Analysis (STPA). By embedding systems thinking into the testing pipeline, SAMT provides:

• Rigorous test adequacy evaluation that captures multi-component interactions
• Automated scenario generation guided by safety principles
• A framework for ADS repair and improvement

The authors acknowledge significant open challenges remain in implementing this paradigm, but position SAMT as an essential step toward more reliable autonomous driving validation.