RiskBlend Prioritizes ML Test Inputs by Combining Four Risk Signals

When machine learning models are updated, previously correct predictions can suddenly fail—a regression fault that's expensive to catch because verification often demands human review or costly simulation. RiskBlend, a new framework submitted to arXiv, tackles this challenge by intelligently ranking test inputs so that teams can verify the highest-risk cases within a limited budget.

The Problem with Regression Testing in ML#

Unlike traditional software testing, ML regression detection can't rely on cheap automated checks. Confirming whether a misclassified prediction represents a genuine model failure requires ground truth verification—something that may involve human annotation, expert judgment, or expensive external validation. Organizations need to prioritize which inputs to check first to maximize the chance of catching real problems with the smallest verification effort.

How RiskBlend Works#

The framework combines four complementary signals to score each test input:

• Historical failure patterns — inputs that failed in past retrainings are flagged as higher risk
• Prediction shift — cases where the model's output changes between versions
• Decision-boundary shift — inputs near regions where the model's decision boundaries have moved
• Neighborhood change — test cases in regions where neighboring predictions have shifted

These four signals are weighted and combined using a validation-learned approach optimized for APFD-squared, a standard metric for measuring how quickly regression tests find faults.

Experimental Results#

Across 1,200 experimental configurations spanning four datasets, five classifiers, and multiple retraining scenarios, RiskBlend achieved the highest average ranking efficiency across all test combinations, with improvements up to 0.32 APFD over existing methods. The approach remained classifier-agnostic, though confidence-based baselines stayed competitive for linear models on sparse categorical features—a gap the researchers attribute to differences in feature-space geometry.

Why It Matters#

By treating ML model changes as a multi-dimensional signal rather than relying on single-model confidence scores, RiskBlend captures behaviors that traditional methods miss. The framework demonstrates that cross-version signals provide essential information for managing the cost and risk of regression testing in production ML systems.