AI
Bringing AI Agents to Production: A Ray-Based Scaling Framework
Scaling agentic AI systems from prototype to production requires solving orchestration, observability, and fault tolerance challenges—and Ray Serve offers built-in infrastructure to handle them.
InfoQ
Running a single AI agent locally is straightforward. Deploying multiple agents in production at scale, handling dynamic traffic, ensuring observability, and maintaining fault tolerance across distributed systems is an altogether different challenge. At InfoQ's QCon conference, engineers from Anduril and the Ray team explored how to architect agentic systems that work reliably at production scale.
The Anatomy of an Agent#
At their core, AI agents combine several key components working in concert. An LLM drives the execution flow—planning actions, orchestrating tool calls, summarizing results, and applying safety checks. Tools grant the agent capabilities, from resource access to data retrieval and external actions. Memory personalizes user experience and seeds contextual knowledge. And the prompt determines behavior through persona, intent clarification, and orchestration directives (guidelines and constraints on what the agent should and should not do).
When running agents at scale, observability becomes critical: teams need visibility into reasoning loops and tracing to ensure the agent behaves as intended. Safety guardrails must be woven throughout the execution lifecycle—from input validation through planning, orchestration, and final synthesis.
From Local Testing to Production Challenges#
The session used a trip-planner agent as a worked example: an LLM wired with tools for destination search, weather lookup, and itinerary synthesis. A developer might start by composing this as a standard Python graph (using popular frameworks like LangGraph or LlamaIndex), then wrap it in FastAPI and test locally. Containerization and Kubernetes deployment follow naturally.
But stateful, multi-turn agents with long-running chains of tool calls quickly expose the limits of a basic RESTful microservice model. Production deployments demand solutions for:
- Capacity and scheduling: Dynamic traffic requires intelligent autoscaling—not just on CPU and memory, but on concurrent request volume, since agents make many parallel network calls. Administrators need granular control over minimum/maximum replicas and preemption policies.
- Fault tolerance: Failures occur at infrastructure (node crashes) and application layers (memory leaks, crashes). Both must be handled transparently.
- Protocol diversity: Beyond HTTP/REST, agents need server-sent events and streaming support for incremental responses, plus APIs for long-running background tasks.
- : Agentic workloads generate dynamic, non-deterministic call graphs. Teams need visibility into LLM token usage, tool invocations, and end-to-end request tracing across complex call chains.
