Framework for Building Reliable Production LLM Agents Through Architectural Pattern Design

Production systems that combine large language models with software infrastructure face a fundamental design challenge: managing the interaction between unpredictable model outputs and deterministic system logic. A new research paper from arXiv tackles this problem by treating that boundary as a core architectural building block.

The Stochastic-Deterministic Boundary#

At the heart of the proposal is the concept of the stochastic-deterministic boundary (SDB) — the point where an LLM's output transitions into a concrete system action. Rather than treating this boundary as an afterthought, the researchers argue it should be a first-class architectural concern with a formal four-part structure: a proposer (the LLM), a verifier, a commit step, and a reject signal that together govern how output becomes action.

Six Core Runtime Patterns#

The paper organizes production agent design around three key dimensions: Coordination, State, and Control. Within this framework, the researchers catalog six distinct runtime patterns that handle the SDB in different ways:

• Hierarchical delegation — suitable for structured task breakdowns
• Scatter-gather plus saga — for parallel task execution with consistency management
• Event-driven sequencing — leveraging event logs for state transitions
• Shared state machine — coordinating multiple agents around common state
• Supervisor plus gate — adding approval layers for critical decisions
• Human in the loop — maintaining human oversight of model outputs

Each pattern traces its origin to established distributed-systems concepts, adapted for the challenge that the "worker" (the LLM) is fundamentally stochastic.

Methodology and Failure Analysis#

The researchers provide a systematic five-step methodology for selecting the right pattern for a given workload, alongside a diagnostic procedure that maps real production failures back to weaknesses in pattern design. They also identify replay divergence — a failure mode where different versions of a model or changes to prompts cause the same deterministic event log to produce different downstream results.