Agentic AI
Building agentic AI systems with Kubernetes: how open-source tools power the learning loop
Open-source models and cloud-native tools are enabling organizations to run complete AI training cycles on a single Kubernetes control plane, from inference through fine-tuning and agent execution.
1/2
The New Stack
The gap between proprietary and open-weight AI models has narrowed significantly. According to the Stanford AI Index 2026, leading closed models held only a 3.3% performance advantage over top open alternatives as of March 2026—small enough that other factors have begun to dominate. Publicly available models like Qwen and GLM-5, distributed through platforms such as Hugging Face under permissive licenses, can now run on open-source inference engines like vLLM and SGLang, giving organizations genuine flexibility.
But serving a downloaded model is just the start. The real value emerges when teams can continuously refine these models using their own data—through fine-tuning, reinforcement learning, and iterative feedback loops. That process is where the complexity, and the cost, lives.
The challenge: three workloads, three separate platforms#
Consider a concrete scenario: a platform team deploys an AI coding agent powered by an open model to automatically fix failing CI/CD builds. When tests fail, the agent clones the repository, analyzes logs, writes a fix, and runs tests to confirm the solution works. Every attempt generates a clean signal: pass or fail.
Those successful repairs create training data. Periodically, the team fine-tunes the model on these examples or applies reinforcement learning against the test results, then evaluates the improved version and deploys it back into service. Over time, the model learns how their codebase typically breaks—and how to fix it.
The problem: this workflow demands three fundamentally different execution patterns. Inference requires long-running, horizontally scaled replicas that start quickly. Fine-tuning and reinforcement learning run the opposite way—they need all-or-nothing GPU allocation across multiple accelerators, completing a single batch and stopping. Agents need ephemeral sandboxes that spin up and down in seconds.
When teams build separate infrastructure for each stage, they inevitably waste resources. GPUs sit idle in one place while jobs queue in another. Every handoff between systems adds friction and cost.
A unified approach: Kubernetes as the foundation#
Rather than assembling a patchwork of specialized platforms, the solution treats the entire learning loop as a standard cloud-native problem. Kubernetes already excels at reconciling desired state with reality and scheduling work across nodes. The key is extending those core primitives to handle models, agents, and accelerators.

