AI Security
AWS Claims Its AI Security Agent Fixes 89% of Vulnerabilities—Here's What That Actually Means
AWS launched Continuum, an automated code-security tool that achieved an 89% success rate on a benchmark test, but the metric masks important limitations about what the system can actually do.
1/2
The New Stack
AWS released Continuum, an AI-powered security agent, to general availability this week. The tool scans code repositories for vulnerabilities, validates them against your AWS environment, and automatically generates pull requests with remediation suggestions.
The announcement came with an attention-grabbing headline: an 89% success rate on an industry benchmark test—significantly higher than previous publicly reported results. However, unpacking what that percentage actually represents reveals a more complicated picture.
The Benchmark and the Numbers#
AWS tested Continuum against CyberGym-E2E, a standardized evaluation framework that asks an AI agent to locate and repair security flaws in open-source software. The system gets 90 minutes to identify a vulnerability in one of 139 projects, with no description, proof of concept, or existing patch as reference material. The agent must then trigger the bug, develop a fix, and ensure the repair doesn't break existing tests.
According to AWS's test results, Continuum successfully detected and reproduced bugs in 92.5% of cases. It then stopped the crash in 89.6% of attempts and maintained passing tests in 89.0%—the figure AWS emphasized in its announcement. However, there's a significant drop-off at the final stage: the patches fixed the specific historical vulnerability the benchmark was designed to test in only 37.8% of cases.
What the Gap Reveals#
The distinction between these two metrics exposes the core limitation. Continuum achieved 89% success by finding and fixing a real vulnerability without breaking the codebase—genuinely useful work. But since the benchmark tests against a specific historical bug that the agent was never informed about, only 37.8% of the fixes addressed that particular flaw.
AWS offers a reasonable defense: repositories often contain multiple security issues, and patching a different vulnerability still provides genuine value. The question AWS doesn't answer is how well Continuum performs when developers point it toward a specific known vulnerability—a more realistic deployment scenario.
The Scope and Self-Testing Factor#
The benchmark focuses exclusively on memory-safety bugs in C and C++, explicitly excluding many other programming languages and vulnerability categories. Additionally, AWS administered its own testing and ran the evaluation internally, with no independent verification of the results.
