New Method Detects Code Vulnerabilities Using AI Without Labeled Examples

A research team has developed an approach to automatically spot security flaws in code by treating vulnerable patterns as statistical anomalies—a shift that sidesteps the traditional reliance on hand-labeled training datasets.

The method, called ANVIL, leverages large language models (LLMs) pre-trained on vast amounts of unlabeled code. Rather than asking an LLM directly "is this code vulnerable?" (an approach that performs only marginally better than random guessing), the researchers instead use the model to reconstruct masked lines of code. Code segments that deviate significantly from what the model predicts become candidates for suspicious behavior.

How It Works#

The core idea rests on a simple premise: vulnerable code is rare compared to typical, well-formed code. An LLM trained on billions of code examples learns what normal code looks like. When presented with a line where a critical portion is hidden, the model generates its best guess for the missing part. If the actual code diverges sharply from that prediction—especially in ways that suggest odd or unsafe patterns—the system flags it as potentially anomalous.

To score these deviations, the researchers combined four signals: whether the reconstructed code matches the real code exactly, the mathematical distance (cross-entropy loss) between the model's predicted and actual tokens, how confident the model was in its prediction, and structural complexity measures. This hybrid scoring approach proved more effective than relying on any single metric.

Testing Against Real Vulnerabilities#

The team evaluated ANVIL across multiple LLM families and configurations, using the PrimeVul dataset as a benchmark. Importantly, they tested against vulnerabilities discovered after the LLMs' training cutoff dates, ensuring the vulnerabilities were genuinely novel to the models.

The results exceeded expectations. ANVIL outperformed established supervised detectors—including LineVul, LineVD, and LLMAO—by significant margins: up to 2x higher accuracy when looking at the top-3 predictions, 75% improvement on a metric tracking missed vulnerabilities, and substantial gains in ROC-AUC score, a standard measure of detection reliability.

Practical Validation#

Beyond benchmark numbers, researchers demonstrated real-world applicability by coupling ANVIL with —tools that automatically generate and test unexpected inputs. This integration helped uncover in existing systems, proving the method finds genuine security issues in the wild.