Modal's Parallel Scheduling Approach Scales to 1M Sandboxes Without Kubernetes

Modal's engineering team has published details on how they redesigned their sandbox infrastructure to handle a scale Kubernetes simply wasn't built for: creating 1 million concurrent sandboxes in under a minute, with median startup times below 500 milliseconds.

Why Kubernetes Hits a Wall at Scale#

Traditional orchestration systems like Kubernetes rely on centralized decision-making and globally consistent state—a fundamental bottleneck when managing millions of containers. The scheduler and its backing datastore (etcd) both experience load that grows with the number of nodes and pods, and frequent writes from multiple sources create contention. Fixing these issues isn't trivial; it demands replacing core components and rewriting the scheduler itself.

At Modal's target scale—tens of thousands of compute nodes creating 50,000 sandboxes per second—these architectural assumptions break down.

A Distributed Approach#

Rather than trying to patch Kubernetes, Modal's engineers made a clean departure. Their key decisions:

• Distributed truth: Each worker node maintains its own resource state instead of deferring to a central datastore
• Parallel scheduling: A fleet of scheduling servers operate concurrently, removing serialization bottlenecks
• Direct RPC: Schedulers contact workers directly to request sandbox creation; workers accept or reject based on available capacity

This model trades global consistency for horizontal scalability, aligning incentives toward speed and isolation over unified coordination.

The Tradeoff#

The architecture consolidates state into a single Redis stream where all workers publish updates—the one identified chokepoint. But testing shows this remains viable up to over 100,000 workers, well beyond current deployment sizes.

Industry Perspective#

AWS's Alex Jones framed Modal's move as symptomatic of a broader shift: Kubernetes excels at shared-memory, multi-agent workflows but struggles with GenAI's need for millisecond-scale isolation boundaries. He suggests the industry is heading toward a split: "The execution plane wants what Modal built; isolation boundaries that appear in milliseconds."