Large Code Models Struggle to Spot Security Vulnerabilities Without Extra Context

Researchers have conducted a comprehensive audit of code language models and their ability to automatically detect security patches, uncovering significant limitations in how these systems identify vulnerability-fixing commits.

The Problem#

Recognizing when a commit addresses a security vulnerability is crucial for rapid patch deployment—particularly since official security advisories typically lag the actual patches by a median of 25 days, and many fixes never receive formal advisories at all. While large language models trained on code are increasingly being deployed for this task, it remains unclear whether they can genuinely recognize a security fix from examining the code change itself, separate from supporting metadata like commit messages or project history.

Methodology and Scale#

The research team unified and re-evaluated 20 fragmented datasets containing over 180,000 commits, creating a single standardized framework for assessment. They trained 270 models across a spectrum of sizes—from 125 million to 80 billion parameters—and systematically isolated the pure code signal by testing on code changes alone, without relying on commit messages or project-level information that could inadvertently leak signal.

Key Findings#

The results are sobering. Model size alone provides only marginal gains: while larger models perform better than smaller ones, these improvements are insufficient to solve the core challenge. When testing with a stringent false positive rate of just 0.5%, every fine-tuned model evaluated—regardless of size—failed to identify at least 80% of actual vulnerability fixes.

Additional context signals—such as information about function internals or broader repository structures—offered no reliable improvement under practical false positive constraints. The researchers also discovered a secondary issue: label quality errors concentrate in commits lacking formal CVE (Common Vulnerabilities and Exposures) associations, which substantially distorts evaluation metrics and inflates apparent model performance.

Implications#

The work suggests that current approaches to vulnerability detection through code language models face fundamental challenges that cannot be overcome simply through scaling or adding contextual information. The authors provide concrete guidance for future research, including recommendations for dataset construction and evaluation methodology when assessing vulnerability detection systems at scale.