AI Architecture
Decoupling AI Workflow Logic From Runtime Unlocks Both Production Durability and Fast Testing
A design pattern lets the same workflow code run reliably in production on persistent runtimes while also iterating quickly through lightweight evaluation loops.
InfoQ
AI systems that chain together multiple steps—including calls to large language models—face a fundamental tension: production deployments demand durability and crash recovery, while development requires fast, ephemeral evaluation loops. Brex's AI platform solved this by writing workflow logic that is completely agnostic to its runtime environment, then plugging in different adapters depending on context.
The Competing Demands#
Building AI workflows requires two kinds of infrastructure that pull in opposite directions. On the production side, systems need what distributed computing has demanded for years: the ability to survive crashes, deploys, and pod recycling. This means every step must be persisted, potentially spread across multiple workers, and replayable if something goes wrong. A long-running agent that spans dozens of LLM calls cannot afford to lose forty minutes of progress because infrastructure was recycled.
Evaluation, by contrast, demands the opposite setup. When tuning prompts or testing branching logic, teams want to load a dataset of examples, run the agent against it, and see scores—all in seconds, locally, in-process. Nothing should be distributed or persistent. Dependencies should be mocked so their outputs stay fixed while only the LLM (the part being evaluated) changes between runs.
These two requirements—heavyweight, durable execution versus lightweight, ephemeral evaluation—are fundamentally at odds. Yet most AI workflow tools force developers to pick one or the other.
The Coupling Problem in Existing Frameworks#
Agent frameworks like LangGraph and Mastra couple orchestration directly to their runtime by design. Control flow is expressed in the framework's SDK or DSL, so the workflow logic and the framework become a single artifact. To evaluate the logic, you run the framework; to serve it, you run the framework. There is no orchestration that exists independently.
Similarly, durable workflow engines like Temporal allow general-purpose code but impose constraints—no direct I/O in the orchestration, determinism requirements, payload limits—that exist to ensure replayability. The orchestration must be written against the engine's rules, not against a generic interface.
This tight coupling creates a chronic problem: eval-prod skew. When logic lives in two different systems (say, one in Temporal for production and another in a test harness for evals), the two versions inevitably drift. A bug in one rarely exists in the other, and deployments can ship behavior that was never validated.
