AI
New Benchmark Shows AI Agents Struggle to Build Static-Analysis Security Checkers
CheckerBench evaluates whether coding agents can autonomously develop security checkers from vulnerability specifications—and finds the task remains largely out of reach.
Researchers have introduced CheckerBench, a new evaluation framework designed to test whether AI coding agents can independently create static-analysis checkers that detect security vulnerabilities. The work reveals that current models perform surprisingly poorly at this long-horizon task, achieving a mean success rate of just 32% despite significant progress in other software-engineering domains.
What the Benchmark Tests#
Building a security checker demands that agents juggle multiple complex steps in sequence: parsing a vulnerability specification, exploring a codebase, writing analyzer-specific detection logic, and iteratively refining the checker as compilation and test feedback arrive. CheckerBench assembles 300 distinct tasks drawn from real-world data—297 common vulnerabilities and exposures (CVEs) sourced across 167 different repositories, 85 weakness categories, and five programming-language ecosystems.
Each task provides agents with vulnerable and patched code versions, a controlled testing environment, and a starter scaffold to build upon. The benchmark itself is executable, allowing rigorous measurement of success.
How Checkers Are Evaluated#
Rather than rely on subjective assessment, CheckerLab—the companion evaluation framework—rebuilds each submitted checker independently and measures four objective dimensions: whether the checker correctly identifies vulnerable code, how precisely it pinpoints the defective location, how many false alarms it generates, and how effectively the agent uses available development tools.
Current Results and Implications#
Testing across 21 different model-harness combinations with three repeated runs per setup showed that the top performers reached roughly 45% success, while the average hovered around 32%. These findings underscore that creating robust, reusable security checkers remains a frontier challenge for AI agents, even as the field has made strides in patch generation and vulnerability spotting elsewhere.
