Study finds specification prompts cut critical defects in LLM-generated backend code by up to 70%

Large language models have stalled in their ability to generate secure code, with defect rates unchanged for four years. A new rigorous study suggests a simple intervention—prepending a detailed specification to the prompt—can meaningfully improve the safety of code generated for regulated backend systems.

The problem: AI code still fails security#

In domains where correctness matters most—financial transactions, healthcare data handling, insurance logic—the same classes of bugs persist in AI-generated code: improper money arithmetic, flawed time handling, unsafe retry logic, and broken access controls. Prior research on instruction files for code generation found little universal benefit, leaving teams searching for more effective guidance strategies.

Testing specification-driven prompting#

Researchers conducted a pre-registered, controlled experiment on 50 realistic backend tasks drawn from real finance, healthcare, and insurance practice. Each task was run through five frontier language models (representing different vendors) under two conditions: with a bare prompt, and with a 267-word specification frame—a fixed preamble laying out what the result must satisfy.

Nine domain-specific, deterministic checkers evaluated the code. The independent Bandit security scanner, which had no knowledge of the specification approach, analyzed outputs separately.

Results: consistent gains across all models#

The specification frame reduced defects across every model tested, cutting between 0.16 and 0.70 findings per task on average. All improvements held statistical significance after multiple-comparison correction, with confidence intervals consistently excluding zero. When the two approaches differed, the specification arm outperformed the bare prompt in 95 of 100 cases and never made any model perform worse in any domain.

Bandit found a stark difference: 53 medium-or-high severity issues in bare-prompt outputs versus just 11 in specification-guided ones, with the advantage consistent across every model.

Why it works#

The effect was strongest where models' unguided defaults were weakest—the specification supplies the discipline a model otherwise lacks. Rather than hoping general instruction files will transfer across tasks, this approach embeds the necessary safeguards directly into each generation request.