AI coding assistants generate insecure infrastructure code without proper guidance, study finds

Artificial intelligence models tasked with writing infrastructure automation code are generating vulnerable systems by default, according to a new study published on arXiv. The research highlights a critical gap in how AI assistants handle security requirements in machine-executable configuration code.

The security gap in AI-generated infrastructure code#

Researchers evaluated 16 different AI models on their ability to generate secure Ansible roles—a common infrastructure automation format—for Apache Tomcat and MongoDB. They tested 278 generated roles against industry security standards from CIS benchmarks, industry baselines for hardened system configurations.

Without explicit security instructions, every single model produced code containing security defects that would leave deployed systems exposed to attack. The generated infrastructure failed compliance checks and performed worse than code written by experienced human developers, raising concerns about the downstream risks of deploying AI-assisted infrastructure automation at scale.

Improving outcomes through structured prompting#

The researchers developed a solution rooted in prompt engineering: they created an extended version of the CO-STAR framework that embeds both Ansible best practices and CIS benchmark requirements directly into the model instructions. Rather than detecting security problems after deployment, this technique aims to prevent them during code generation.

The results were mixed but promising for capable models. When using this structured prompting approach:

• 4 of the 16 models produced code that met compliance requirements
• The leading performer achieved 95%-100% CIS compliance—roughly four times better than the 23%-43% baseline for human developers
• Overall code quality improved by 19%-49% across different metrics

Why some models still fail#

The remaining 12 models continued to generate non-compliant code, but not because they lacked the underlying capability. Instead, they struggled to follow instructions containing multiple constraints simultaneously. This suggests that model architecture and instruction-following ability, rather than domain knowledge, represent the bottleneck.

The practical advantage of this finding is significant: the approach requires no model retraining. Developers can adopt the structured prompting technique through system-level instructions, making it accessible immediately without waiting for new model versions or fine-tuning.