Runtime Data Boosts LLM Accuracy in Predicting Jupyter Notebook Crashes

Machine learning researchers frequently work in Jupyter notebooks for rapid prototyping and experimentation, but the interactive environment makes notebooks vulnerable to crashes that disrupt workflow and slow development. A new approach shows that feeding runtime information to large language models can substantially improve their ability to spot and diagnose these failures before code executes.

The Problem with Notebook Crashes#

Interactive notebooks enable flexible, iterative ML work—but they also make crashes harder to predict. When a cell depends on specific data states or library configurations from earlier cells, problems may not surface until execution. Current techniques for catching errors in notebooks rely primarily on static code analysis, missing the broader context that runtime state provides.

CRANE-LLM's Hybrid Approach#

Researchers developed CRANE-LLM, a system that combines source code inspection with structured runtime data extracted directly from the notebook kernel. The method feeds both elements to state-of-the-art LLMs (Gemini, Qwen, and GPT-5) to predict whether a target cell will crash and, if so, to diagnose the underlying cause.

Performance Gains from Runtime Context#

Testing on JunoBench—a benchmark of 111 real Kaggle ML notebooks with verified crashes across multiple libraries and error types—revealed significant improvements. Compared to analyzing source code alone, adding runtime information lifted accuracy by 7–10 percentage points and F1-score by 8–11 points. The gains were particularly strong for diagnosis tasks, suggesting runtime context helps models reason about why crashes occur, not just whether they will happen.

Why This Matters#

As Jupyter notebooks become central to ML development workflows, tools that catch errors early can save hours of debugging. The findings indicate that LLMs benefit substantially from access to dynamic execution state—a principle that could extend beyond notebooks to other interactive or stateful coding environments.