LLM-Generated Code Poses Major Security Risks in Real-World Applications

Researchers have raised alarm bells about the safety of "vibe coding"—a development paradigm where software engineers use large language model (LLM) agents to handle complex coding tasks with minimal oversight. Despite its growing popularity, a new study suggests that code generated by these AI systems contains significant vulnerabilities that could compromise production systems.

The SUSVIBES Benchmark Study#

A team of researchers introduced SUSVIBES, a benchmark dataset comprising 186 real-world feature requests drawn from open-source projects—cases where human developers had previously implemented code with known security flaws. The researchers then tested 12 different LLM-powered coding agent configurations using state-of-the-art models to see if they would reproduce or avoid these same vulnerabilities.

Alarming Security Track Record#

The findings were sobering. While the most advanced setup—SWE-Agent paired with Claude 4 Sonnet—achieved a 57% functional correctness rate, only 11.8% of those solutions were actually secure. This stark gap between what works and what's safe underscores a fundamental problem: AI agents can generate code that runs correctly but harbors exploitable weaknesses. Across all 12 agent configurations tested, security performance lagged far behind functional performance.

Limited Effectiveness of Current Mitigations#

The research team explored whether adding vulnerability hints to feature requests could help. These preliminary security strategies failed to meaningfully reduce the security issues, suggesting that the problem runs deeper than a simple lack of context or awareness during code generation.

Implications for Production Deployment#

The results carry serious implications for teams considering vibe coding in security-sensitive domains. As LLM-assisted development becomes more widespread, the mismatch between correctness and safety poses a hidden risk—code that appears to work but contains exploitable flaws could slip into production undetected.

The researchers have made their benchmark, dataset, and leaderboard publicly available to support continued investigation into the security characteristics of agent-generated code.