Kubernetes
Uber Decouples Scaling Decisions From Execution to Enable Safe Multi-Orchestrator Kubernetes Management
The platform company introduced ServiceScale to let multiple controllers manage scaling intent independently, solving the challenge of regional failover without maintaining reserved idle capacity.
InfoQ
Uber has unveiled how it tackled a critical scaling problem on its massive Kubernetes fleet: letting multiple systems—such as a failover orchestrator and the primary deployment controller—safely influence the same scaling decisions without stepping on each other's toes.
The Challenge: Failover Without Idle Capacity#
The company operates active-active data centres across regions and historically maintained reserved idle compute in all locations to absorb traffic surges during outages. When a region fails, remaining sites must have enough spare capacity to handle the redirected load. Uber wanted to be smarter: instead of paying for perpetual idle reserves, the platform could automatically scale down lower-priority workloads during a failover and scale up critical services to use that freed capacity.
This created a novel problem. The company's primary deployment system (Up and its Uber Deployment Controller) owned the steady-state scaling goals, but now a failover orchestrator needed to temporarily override those goals. Adding failover logic directly to the existing controller risked introducing instability into the most sensitive part of the fleet—any bug could break both normal operations and emergency failover across thousands of services.
Separating Intent From Execution#
Rather than bake failover decisions into the main controller, Uber created a new abstraction: the ServiceScale custom resource and a dedicated Service Scale Controller (SSC). Each orchestrator—whether managing steady state or responding to a failover—expresses its desired scaling targets through ServiceScale. The SSC then combines all those inputs and translates the final intent into actual Kubernetes objects like ReplicaSets.
The design kept things deliberately simple. According to the engineers behind the system, they avoided "an additional external database, a separate coordination service, or a control plane that'd become harder to debug under incident pressure." By storing all scaling intent directly in Kubernetes custom resources, operators could inspect the state by examining ServiceScale objects—seeing exactly which orchestrator wanted what scaling level—and recovery became straightforward since both permanent and temporary failover data lived in the same spec.
