LLM agents can slash security testing false alarms, but results vary wildly by model and vulnerability type

Developers working with static application security testing (SAST) tools face a persistent headache: these scanners catch real vulnerabilities but also generate mountains of false positives, forcing teams to waste time on unverified alerts. A new study from arXiv researchers explores whether large language model (LLM) agents—systems that can reason iteratively, use tools, and interact with their environment—can help separate the signal from the noise.

Testing three major agentic frameworks#

The researchers benchmarked three state-of-the-art LLM-based agent architectures—Aider, OpenHands, and SWE-agent—against SAST false positives. They evaluated the systems on two datasets: the OWASP Benchmark (a standard testing ground for security tools) and real-world Java repositories from open-source projects, focusing on CodeQL alerts.

Strong results, with important caveats#

The findings were striking in their potential: across the OWASP Benchmark, the best agent configurations reduced the false positive rate from over 92% down to just 6.3%. On real-world code, the top configurations identified false positives with up to 93.3% accuracy. Those numbers suggest LLM agents could genuinely help developers triage alerts more efficiently.

But the study reveals significant trade-offs:

• Model strength matters: Stronger backbone models like Claude Sonnet 4 and GPT-5 show dramatic gains when paired with agentic frameworks compared to simple prompting. Weaker models see little or inconsistent improvement.
• Vulnerability category is crucial: The benefits of using agents shift depending on the type of security issue being evaluated.
• Cost is a concern: Different agent frameworks carry vastly different computational costs, which could affect real-world adoption.
• True positives are at risk: Aggressively filtering false positives sometimes suppresses legitimate vulnerabilities, creating a difficult operational trade-off.

What this means for security teams#

The research demonstrates that LLM agents represent a genuinely powerful but uneven solution to the false positive problem. Deploying them effectively requires careful choices about which agent architecture to use, which underlying language model to select, and how aggressive to be with filtering. There is no one-size-fits-all answer.