Automated Testing Tool Helps Catch Silent Failures in Machine Learning Notebooks

Jupyter notebooks have emerged as the standard environment for data scientists to prototype and iterate on machine learning projects, combining code, data, and visualizations in a single interactive interface. Yet this convenience comes at a cost: notebooks typically lack robust testing infrastructure, allowing subtle bugs and regressions to slip through undetected during active development.

The Problem: Silent Failures in Notebooks#

Without formal testing mechanisms, changes to data processing pipelines, model training logic, or evaluation procedures can produce incorrect results without crashing or raising an error. These "silent regressions" are particularly dangerous in ML work, where outputs may appear reasonable on the surface while being fundamentally broken. The consequence is that notebook-based experiments become difficult to reproduce and trust over time.

Introducing NBTestGen and NBTest#

Researchers have introduced NBTestGen, an automated system for generating regression-based assertions tailored to ML workflows. Rather than requiring manual test writing, NBTestGen analyzes notebook cells and produces assertions that verify properties of data transformations, model construction, and evaluation procedures. The generated assertions are powered by a new regression testing framework called NBTest, which operates as a Jupyter plugin and lets developers attach cell-scoped tests directly to notebook code.

These cell-scoped assertions offer three key advantages: they execute only after their corresponding cells run, they allow notebooks to continue functioning when tests fail (rather than halting execution), and they integrate cleanly with pytest and continuous integration pipelines for teams managing notebooks in version control.

Validation at Scale#

The approach was evaluated on 585 notebooks from Kaggle. NBTestGen generated approximately 21,500 assertions overall (roughly 37 per notebook on average). In testing against intentional code mutations designed to simulate real bugs, the generated assertions detected 72% of ML-specific faults while maintaining a 100% pass rate on clean code. When applied to 531 older versions of Kaggle notebooks, the assertions caught nearly 70% of actual historical regressions that had gone unnoticed.

The SHAP library—a popular tool for explaining machine learning models—has already integrated NBTest into its CI infrastructure. A user study with 17 ML developers found that teams perceived NBTest as intuitive and valuable for their workflows.