How Amazon EKS Reinvented Kubernetes Infrastructure for AI-Scale Workloads

Amazon EKS operates hundreds of thousands of Kubernetes clusters across more than thirty regions worldwide. Running at that magnitude has yielded a crucial insight: most availability issues don't arise from individual component breakdowns, but from how those components respond to problems. A stale cache returning incorrect data or a health check that terminates a critical process can transform a minor fault into a full outage. The difference between a robust control plane and a fragile one lies not in eliminating faults—it's in stopping them from multiplying.

The New Demands of Modern Workloads#

Traditional Kubernetes was designed for predictable, measured change: pods launching and stopping at steady rates, controllers reconciling over seconds or minutes. That assumption no longer holds.

Foundation model training now scales across thousands of GPU nodes in minutes. Real-time inference services ramp from baseline capacity to thousands of replicas in an hour, then contract just as quickly. Apache Spark jobs burst from zero to tens of thousands of executor pods, process data, and vanish. Agentic AI systems introduce another layer: autonomous agents that spin up, branch into parallel tasks, execute, and tear down in seconds or less.

These workloads compress enormous volumes of state changes into tiny time windows and tolerate no delays. That velocity forced AWS to reimagine Kubernetes' core mechanics to handle scale that had previously seemed impossible.

Rethinking the Storage Foundation#

Every Kubernetes cluster depends on etcd as its single source of truth. Losing etcd means losing the entire cluster's memory. At the scale EKS operates, protecting that state required fundamental rethinking.

AWS deployed an operator agent alongside every etcd instance to automate its entire lifecycle. The agent handles two critical tasks: snapshot-based backup and recovery (restoring from snapshots if a majority of instances are lost) and membership management (safely replacing failed members without risking split-brain failures).

A larger breakthrough came next: replacing etcd's Raft consensus mechanism with a purpose-built durable journal. In standard etcd, a majority of members must agree on every write before it commits. Lose two of three nodes, and the cluster goes offline.

By moving durability to an external journal, etcd members no longer need to negotiate quorum. Writes commit as soon as the journal persists them, eliminating an entire class of quorum-loss failures. Since the journal handles durability, etcd stopped fsyncing to local disk and moved its data store to memory. What was a disk-bound bottleneck became compute-bound, removing storage latency from the critical path entirely.