Loading post…
ShopifySam ObeidExplainer
Running Apache Kafka on Kubernetes at Shopify
Summary
Shopify adopted Apache Kafka in 2014 as a data bus for reliable messaging, initially collecting events and aggregating logs, then expanded to multi-tenant regional and aggregate clusters delivering billions of messages daily. In 2017, it migrated Kafka infrastructure from data centers to the cloud with zero downtime by mirroring regional data to both aggregate clusters before moving producers and consumers to cloud clusters. Shopify chose Kubernetes over GCP virtual machines because StatefulSets provide ordered, graceful deployment and scaling, while Custom Resources and Controllers support cluster-specific automation. The implementation uses self-built Kafka images in an own registry, persistent volumes, node and inter-pod affinity controls, taints, rack-awareness across zones, readiness probes, and resource-monitoring sidecars to limit restart and availability risks. The migration completed successfully, and Shopify now runs multiple regional clusters plus an aggregate cluster that mirrors data for the data warehouse.
Context
Shopify moved Kafka from Chef-managed data-center infrastructure to cloud-based Kubernetes while maintaining service availability. The system includes regional clusters and an aggregate cluster that mirrors data for the data warehouse.
Approach / What changed
The migration deployed regional Kafka clusters in each cloud region and an aggregate cloud cluster, mirrored regional data to both old and new aggregate clusters, then redirected producers and consumers to the cloud. Kubernetes StatefulSets manage ordered deployment and scaling, while Custom Resources and Controllers automate topic provisioning, cluster discovery, and SSL certificate distribution. Shopify also builds and hosts its own Kafka image and uses storage, scheduling, placement, readiness, and rack-awareness controls.
Takeaways
Mirroring regional clusters to both data-center and cloud aggregate clusters allowed Shopify to move clients to the cloud before safely shutting down the old infrastructure.
Kubernetes StatefulSets provide ordered, graceful deployment and scaling for Kafka, while Custom Resources and Controllers add automation for topics, discovery, and SSL certificate distribution.
Kafka broker restarts can trigger terabytes of data shuffling; persistent volumes, anti-affinity, taints, readiness probes, and rack-awareness reduce availability risks.
FAQ
Why did Shopify choose Kubernetes instead of GCP virtual machines for Kafka?
GCP virtual machines would have required Shopify to build tooling for cluster-wide state management, monitoring, and availability-safe deployments. Kubernetes StatefulSets provided ordered and graceful changes for a stateful Kafka cluster, while Custom Resources and Controllers enabled additional behavior.
How did Shopify migrate Kafka to the cloud without downtime?
Shopify deployed regional cloud clusters and an aggregate cloud cluster, then mirrored data from regional clusters to both aggregate locations. After moving producers and consumers to the cloud clusters, it shut down the data-center clusters once they had no incoming traffic or readers.
What happens when a Kafka broker restarts in this setup?
Restarting a broker or container can cause terabytes of data to shuffle across the cluster, and restarting many brokers simultaneously risks offline partitions and data loss. Shopify used persistent volumes, placement controls, readiness probes, and rack-awareness to reduce these risks.
Source and freshness
- Source
- shopify.engineering
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.