New approach uses multiple simulators together to find more reliable autonomous driving bugs

A team of computer scientists has proposed a novel testing methodology called MultiSim that tackles a persistent problem in autonomous driving system validation: the fact that simulators often disagree with each other on whether a given scenario causes failure.

The simulator reliability challenge#

When researchers test advanced driver-assistance systems (ADAS) using driving simulators, they frequently encounter inconsistencies. A scenario that triggers a system failure in one simulator might produce no issues in another—not because of genuine system defects, but due to differences in how the simulators model physics or other implementation variations. These inconsistencies, sometimes called "flakiness," make it difficult to identify which failures represent actual vulnerabilities versus which are merely artifacts of particular simulation tools.

How MultiSim works#

Rather than running test scenarios on a single simulator or combining results from independent testing campaigns, MultiSim evaluates candidate scenarios on multiple simulators simultaneously. The approach uses search-based testing to explore the space of possible driving scenarios while prioritizing those that produce consistent outcomes across the simulator ensemble. Scenarios that fail consistently across different simulation environments are treated as genuine defects worthy of further investigation, while scenarios that fail in only some simulators receive lower priority—a signal they may reflect simulator-specific quirks rather than true system flaws.

Superior performance across benchmarks#

The empirical evaluation involved testing three lane-keeping ADAS using various combinations of three popular simulators. The results demonstrate substantial gains: MultiSim achieved 66% more simulator-agnostic failures on average compared to single-simulator approaches. When stacked against competing multi-simulator methods that combine results from separate test generation runs, MultiSim identified up to 3.4 times more genuine cross-simulator failures and maintained higher failure rates overall.

Efficiency improvements through prediction#

Running tests on all simulator combinations can be computationally expensive, particularly when simulators disagree. The researchers addressed this by developing a surrogate model to predict which test inputs will trigger disagreement between simulators, allowing the system to skip costly redundant executions. This optimization preserves the number of valid failures discovered while improving overall testing efficiency.