Study reveals best metrics for evaluating software test suites in combinatorial testing

Combinatorial interaction testing (CIT) has become an important tool for catching software bugs caused by interactions between different parameter values. Yet practitioners often struggle to decide which evaluation metric to use when assessing the quality of their test suites—a choice that can significantly affect testing outcomes.

The Challenge of Test Suite Evaluation#

Selecting the right evaluation metric is not straightforward. The field offers numerous options, each with its own strengths and trade-offs. Without clear guidance, teams may choose metrics that don't align with their actual testing goals or may waste computational resources on unnecessarily complex approaches. Until now, no systematic study had tackled this decision-making problem head-on.

A Comprehensive Empirical Study#

Researchers conducted an extensive analysis covering eight real-world open-source projects, examining 32 test scenarios and more than 295,000 combinatorial test suites. They surveyed all available black-box evaluation metrics—defining them rigorously, grouping them by type, and analyzing their computational complexity. The team then measured how well each metric predicted whether a test suite would actually detect software faults.

Key Findings#

The research revealed that Value Combination Coverage (VCC) emerges as a reliable predictor of fault-detection effectiveness. This metric can help teams assess test-suite quality with reasonable accuracy.

Another important insight: distribution-based metrics generally run faster than those focused on interaction coverage, making them attractive when computational speed matters. However, test suites that vary significantly in their characteristics may respond differently to different metrics—meaning one-size-fits-all recommendations have limits.

Practical Takeaway#

The study provides actionable recommendations for CIT practitioners choosing between evaluation approaches. The key is matching the metric to both your computational constraints and the specific properties of your test suite, rather than defaulting to the most popular or theoretically elegant option.