Large language models show promise but stumble on pinpointing malicious Python code

Researchers evaluated how well large language models can spot malicious software in Python packages—a growing vulnerability in open-source software development where attackers distribute compromised code through public repositories. The findings reveal significant strengths and meaningful limitations in using AI for this security task.

The Challenge#

Open-source package ecosystems like PyPI face an escalating threat: attackers increasingly disguise malicious code within seemingly legitimate packages that developers download and install. While LLMs have shown promise for code analysis, their ability to reliably catch these threats—and more importantly, explain why code is dangerous—remained unproven.

Study Design and Results#

Researchers tested 13 different LLMs against a dataset of 4,070 PyPI packages (370 malicious, 3,700 benign). They designed two evaluation tasks: first, detecting whether a package was malicious at all; second, pinpointing the specific lines of code and behaviors that make it dangerous.

For the broader detection task, LLMs achieved F1 scores ranging from 0.40 to 0.99, meaning they successfully identified most malicious packages but also generated frequent false alarms by flagging legitimate code as suspicious. Performance deteriorated sharply on the second task. When asked to identify specific malicious behaviors, LLMs managed a weighted F1 score of 0.69; when asked to locate the exact harmful code snippets, that dropped to just 0.48.

Where LLMs Fail#

The analysis revealed critical blindspots. While LLMs recognize common, well-documented attack patterns, they struggle with behaviors that require understanding context beyond isolated code chunks or inferring the developer's true intent. Problematically, models also report malicious indicators that don't actually exist in the code—introducing noise into security workflows.

Testing various prompt strategies, model sizes, context windows, and temperature settings showed that only one factor meaningfully affected performance: code complexity. Longer and more intricate packages became significantly harder for LLMs to analyze accurately.

Practical Implications#

Rather than automating malware detection, researchers recommend deploying LLMs as a first-stage triage filter to flag potentially risky packages for human security experts to review. This hybrid approach leverages the models' strength in broad pattern recognition while sidestepping their weaknesses in detailed, context-dependent analysis.