LLMs Write Infrastructure Code Far Less Securely Than Humans, Study Finds

Researchers have published the first rigorous security comparison between Infrastructure-as-Code (IaC) written by large language models and templates authored by human engineers. The findings paint a stark picture: across matched deployment scenarios, LLMs generate code with security vulnerabilities at rates 3–4 times higher than human baselines.

The Benchmark Design#

The study, termed GenIaC-SecBench, examined 100 deployment scenarios across twelve model configurations from six different vendors, testing both open-source and proprietary systems. Crucially, rather than simply reporting vulnerability counts for models in isolation (the standard in prior work), the researchers scanned 634 real-world human-authored IaC templates with the identical scanning toolchain. This human baseline allows for meaningful comparison—answering the critical question: worse than what?

The team deployed three independent security scanning engines—Checkov, Trivy, and KICS—at complete coverage, yielding 1,196 artifacts and comprehensive vulnerability data.

A Confounding Discovery: Size Matters#

Before drawing conclusions, the researchers identified a critical statistical trap. Vulnerability density proved inversely correlated with code artifact size (Spearman ρ = −0.55, p < 10⁻⁷⁷). Without accounting for this relationship, simply comparing vulnerability counts would measure code size, not actual security quality. By applying size-matching techniques, they eliminated this confounder and obtained defensible results.

The Central Finding#

Size-matched comparison reveals a troubling pattern: every tested model configuration exceeds the human security baseline, generating vulnerabilities at 3.21× to 3.87× the rate of human-authored IaC. The disparity widens dramatically with task simplicity—reaching 4.9× for single-resource deployments but narrowing to 1.4× for scenarios involving twenty or more resources, suggesting LLMs struggle proportionally more when configurations are stripped to essentials.

Filtering for Explicit Insecurity Requests#

The researchers stratified results by prompt class and discovered that many test scenarios prescribed insecure configurations rather than asking only for functional code. When excluding every scenario that explicitly requested insecure defaults, all tested LLM configurations still exceeded the human baseline, albeit by a reduced margin of 2.4× to 4.2×, indicating the vulnerability gap cannot be attributed solely to adversarial prompts.