From One Horizontal Scaling Controller to Many: Evolving Uber's Compute Platform

Summary

Uber’s failover redesign exposed a limitation in its long-standing single-controller model for scaling stateless workloads. To reuse idle capacity during regional failover, the platform needed multiple orchestrators to influence replica counts without overloading UDC, which remained on the normal deployment lifecycle path. The solution introduced the ServiceScale CRD and Service Scale Controller, allowing separate actors to record steady-state and temporary failover intent in Kubernetes and letting SSC reconcile the combined decision. Production work then exposed risks from informer caches that lagged reality and near-simultaneous multi-writer updates, including ReplicaSet metadata-spec drift that disrupted proportional scaling and could leave workloads stuck. Uber added fleet-wide detection, an automated UDC healer, a long-term scaling-path fix, and extensive kind integration tests, staging, and canaries; the year-long rollout avoided customer-impacting outages while establishing a more extensible control plane.

Takeaways

1. 1

   ServiceScale materializes each orchestrator’s scaling intent in Kubernetes, making competing requests inspectable and allowing failback to converge from persisted steady-state and temporary failover data.

2. 2

   Informer caches can lag reality, and near-simultaneous writes can produce ReplicaSet metadata-spec drift that breaks proportional scaling and can leave workloads stuck.

3. 3

   The rollout used kind integration tests, staging, canaries, fleet-wide drift detection, and an automated UDC healer to manage race-condition and production risks.