Agentic LLMs With Active Retrieval Hit 62% Success on Infrastructure Code Tasks

Generating Infrastructure-as-Code (IaC) from natural language descriptions demands more than syntactic correctness—it requires satisfying cloud provider schemas, managing component dependencies, and adhering to organizational policies. A new empirical study evaluates seven agentic strategies for automating Terraform code generation and identifies which techniques work best.

The Testing Framework#

Researchers benchmarked agentic language models on IaC-Eval v2, a 186-task evaluation dataset focused on AWS and Terraform configurations. Rather than simply checking whether code is syntactically valid, the evaluation uses a multi-stage verifier approach: terraform validate (syntax checking), terraform plan (dependency and resource resolution), and opa eval (policy compliance via Rego v1 intent policies). This staged breakdown reveals exactly where generation fails and why.

Key Findings on Strategy Performance#

The study identified five major insights:

• Active retrieval dramatically boosts weak models. Adding Retrieval-Augmented Generation (RAG) via ReAct agents with either Model Context Protocol (MCP) or ChromaDB backing raised Qwen2.5-Coder 7B from 14.0% to 45.7% pass@1 success rate, primarily by cutting validation failures from 144 to 66 tasks (p<0.0001).

• Iterative refinement with verifier feedback works well but shows binary behavior. Tasks refined with verifier error feedback reached 62.9% pass@1 for Qwen 7B and 84.4% for GPT-4o, but exhibited a stark pattern: most tasks either resolved on the first retry or exhausted the retry budget with no middle ground.

• Lightweight prompt optimization improves accuracy without retraining. The GEPA method, which uses reflective instruction optimization trained on only 80 verifier-guided rollouts, boosted the Active RAG baseline by 7.5 percentage points (p=0.026)—evidence that prompt tuning can enhance verifiable code generation without model weight updates.

• Demonstration injection matches RAG without infrastructure overhead. SIMBA, a teacher-free approach that injects demonstrations directly into prompts, achieved performance equivalent to Active RAG (p=1.0) while eliminating the need for separate retrieval systems. However, it could not address the most common failure pattern—self-defined property errors, which account for 50% of remaining failures.