AI Security
Domain-Specific Prompts Don't Actually Improve AI Code Security, Study Finds
A benchmark of 7 large language models across critical infrastructure sectors reveals that industry-framed prompts produce no statistically significant security benefit over neutral baselines.
Researchers have released a comprehensive study challenging the assumption that tailoring AI code generation prompts to specific industry contexts—such as critical infrastructure—produces more secure output. Their analysis of over 5,000 code samples suggests that model selection matters far more than prompt framing when it comes to vulnerability rates.
The SecDrift Study#
The work, titled SecDrift, measured what researchers call "sector-conditioned security drift": how vulnerability rates in AI-generated code change when prompts reference specific industry contexts versus generic ones. The team tested seven large language models (six of which produced code suitable for analysis) across eight CISA-designated critical infrastructure sectors and nine different weakness categories (CWE types), conducting 5,355 separate evaluations.
The researchers used a careful experimental design that held the underlying coding task constant while swapping in industry-specific language. This approach isolated the effect of domain framing from other variables.
Surprising Results#
The headline finding defied intuition: code generated with industry-conditioned prompts showed a vulnerability rate of 14.0%, compared to 11.4% for neutral prompts. However, this apparent improvement was not statistically significant (p = 0.24), and further analysis revealed it was entirely driven by two specific vulnerability types (CWE-502 and CWE-22). Once those were excluded, the industry-framed prompts actually performed slightly worse.
Testing this pattern on non-critical sectors (e-commerce, online education) as a placebo control produced nearly identical results to the critical infrastructure findings, suggesting the observed effect is generic to industry framing itself, not tied to the sensitivity of critical systems.
What Actually Matters#
The study's clearest takeaway: choose your model carefully. Among the full-output models tested, vulnerability rates ranged from 11.6% to 16.1%—a span far larger than any difference prompt framing could produce. These model-level differences persisted consistently across all conditions and sectors.
The researchers released their evaluation framework, prompts, generated code samples, human validation results, and analysis scripts publicly, providing a foundation for future work on AI code security.
