Multi-agent framework improves LLM code reliability through iterative execution validation

Large language models excel at generating plausible code, yet traditional single-pass approaches offer little assurance that the output will actually work. Researchers have developed ExecuGraph, a multi-agent architecture that makes execution-based testing a core part of the code generation pipeline to enhance reliability.

The Framework in Action#

ExecuGraph orchestrates six specialized roles—Planner, Code Generator, Logical Reviewer, Evaluator, Optimizer, and Explainer—using a typed workflow managed by LangGraph. The system runs locally with Ollama-hosted models and includes an optional retrieval component to help agents recall algorithmic patterns. Critically, every code evaluation runs in an isolated subprocess sandbox with strict time limits to prevent runaway execution.

The architecture operates within a bounded retry budget, allowing the multi-agent ensemble to iterate toward correctness rather than gambling on a first attempt. This stands in sharp contrast to the conventional "generate once and hope" approach that most LLM code tools rely on.

Testing Results and Insights#

Researchers tested ExecuGraph on three evaluation suites: an internal 30-problem dataset focused on data structures and algorithms (internal-30), HumanEval (64 problems), and an introductory subset of APPS. They compared the multi-agent system against two baselines: a standard one-shot model and a single-agent setup with retry logic similar to Reflexion.

On the internal-30 suite, performance across all three conditions proved statistically equivalent (p=0.59). However, HumanEval showed modest gains of 3.1 percentage points for the full multi-agent approach. The most striking finding emerged in cross-model experiments: when paired with DeepSeekCoder V2 Lite, accuracy jumped from 57.5% (one-shot) to 80.0% (multi-agent), a gain of 22.5 percentage points.

Why This Matters#

The results suggest that multi-agent decomposition delivers the greatest value when the underlying language model is already capable. This scaling hypothesis implies that as base models improve, the structured reasoning and iterative validation provided by ExecuGraph becomes increasingly powerful. The framework also offers methodological value: a unified codebase that collapses into different configurations (one-shot, retry, per-agent) makes it possible to isolate and measure each component's contribution independently.