Lab benchmarks don't predict real-world success for AI vulnerability detectors

Researchers have exposed a significant chasm between how well vulnerability-detection AI performs in academic settings and how it behaves in the field. While deep learning models and large language models show impressive results on curated benchmark datasets, they stumble badly when faced with actual vulnerabilities in real codebases.

The Gap Between Theory and Practice#

A team of researchers evaluated two representative deep learning systems—ReVeal and LineVul—alongside four state-of-the-art language models (Claude 3.5 Sonnet, GPT-o3-mini, GPT-4o, and GPT-5) on a newly constructed dataset called VentiVul. This dataset differs from typical academic benchmarks in one crucial way: it comprises 20 recently patched vulnerabilities from the Linux kernel, captured in May 2025. These are real bugs that developers actually fixed, not synthetic or carefully curated examples.

The results were stark. When models trained and evaluated on standard benchmarks—Juliet, Devign, BigVul, and ICVul—were tested on VentiVul, their performance plummeted. Most systems failed to reliably detect the actual vulnerabilities, revealing what researchers call a "persistent gap between academic benchmarks and real-world deployment."

Why Models Fail to Generalize#

The underlying problem stems from how these models learn and represent code. Using visualization techniques like t-SNE, researchers found that current models struggle to identify consistent patterns that distinguish vulnerable code from safe code, even within their training distributions. Graph neural networks and transformer-based models learn features that work well on the specific datasets they train on, but these learned patterns don't transfer when the data distribution shifts—which happens constantly in production environments.

Different benchmark datasets have their own idiosyncrasies: inconsistent labeling practices, data from different eras, and different categories of vulnerabilities. When a model trained on one dataset encounters code from another source, its accuracy drops sharply. The situation only worsens with real-world code that has no labeled patterns at all.

The Road Ahead#

The study emphasizes that robust vulnerability detection requires both more reliable code representations and higher-quality datasets that reflect the messiness of production software. The authors' evaluation framework, explicitly designed for deployment scenarios rather than pure academic performance, offers a path forward for testing whether security tools can actually work in practice.