---
title: "Kubernetes"
description: "150 posts about Kubernetes, summarised, each linking to the original."
---

# Kubernetes
> 150 posts about Kubernetes, summarised, each linking to the original.

## Articles

### [Plumbing At Scale](https://yomu.fyi/post/plumbing-at-scale.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Karan Kamath
- Published: Jan 6, 2020
- Type: Problem & solution

Grab's backend services process terabytes of data ingress per hour, generating recurring needs for stream transformations, joins, and time-windowed aggregations across diverse workloads. To support these asynchronous processing patterns across their Go ecosystem, the Coban team developed a managed, NoOps event sourcing and stream processing platform. The architecture packages stateless processing pipelines as Kubernetes deployments on AWS, polling Kafka event logs and using ScyllaDB as a shared metastore for stateful needs like deduplication and windowing. Stream processing pods combine ingestion triggers, a worker pool runtime, and user-provided domain logic plugins with customizable failure handling. This infrastructure scales to handle over 300 billion events weekly while maintaining workload isolation and elastic autoscaling.


### [How Shopify Manages Petabyte Scale MySQL Backup and Restore](https://yomu.fyi/post/how-shopify-manages-petabyte-scale-mysql-backup-and-restore.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 1, 2019
- Type: Problem & solution

Shopify manages a petabyte-scale fleet of MySQL servers across replica-sets, or shards, in three Google Cloud Platform regions, where file-based Xtrabackup archives made shard restores take more than six hours. To reduce that Recovery Time Objective, the team built Kubernetes CronJobs around the Compute Engine Persistent Disk snapshot API, scheduling snapshots every 15 minutes while accounting for regions, zones, instance roles, and MySQL consistency variables. Persistent Disk snapshots take about 20 minutes initially and typically under 10 minutes incrementally; restoring the latest snapshot to a new volume and starting MySQL, including InnoDB instance recovery and replication-lag recovery, typically brings RTO below 30 minutes. Retention tooling keeps the latest two snapshots plus policy-defined dailies and weeklies, while verification checks startup, GTID auto-positioning, and InnoDB page corruption; offsite tooling compresses, encrypts, and transfers exported data to another provider.


### [Catwalk: Serving Machine Learning Models at Scale](https://yomu.fyi/post/catwalk-serving-machine-learning-models-at-scale.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Nutdanai Phansooksai
- Published: Jul 2, 2019
- Type: Problem & solution

As machine learning adoption expanded at Grab, individual teams created fragmented model serving solutions that duplicated engineering effort and required data scientists to handle underlying infrastructure. To resolve these inefficiencies, Grab developed Catwalk, a self-service machine learning model serving platform. The system runs TensorFlow Serving containers across a managed Kubernetes cluster integrated with Grab's observability stack. Data scientists deploy or update models simply by saving files using the tf.saved\_model API to dedicated Amazon S3 buckets, while Kubernetes automates orchestration, ingress routing, and pod autoscaling. Catwalk abstracts server management away from data scientists, shortens deployment timelines, and provides high availability during model version rollouts.


### [A New Kubectl Plugin for Kubernetes Ingress Controller ingress-nginx](https://yomu.fyi/post/a-new-kubectl-plugin-for-kubernetes-ingress-controller-ingress-nginx.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: May 1, 2019
- Type: Announcement

Shopify contributed a kubectl plugin to ingress-nginx, an open-source Kubernetes Ingress controller built on NGINX, because debugging and upgrading the heavily used controller often required long, frustrating sequences of kubectl commands. The plugin’s lint subcommand identifies ingress configuration that may need changes for a new release, while richer ingress listings expose individual paths, target services, and TLS configuration. It also wraps common kubectl operations, automatically selects an ingress pod, and helps inspect generated nginx.conf and dynamic configuration managed with openresty lua-nginx-module. The author reports substantially faster upgrades and debugging, comparing a rollout to version 0.24.1 with an earlier upgrade to 0.22.0 that took days, while acknowledging that growing Kubernetes expertise also contributed. Released with ingress-nginx 0.24.0, the plugin should also be compatible with 0.23.0.


### [Engineering a Historic Moment: Shopify Gets Ready for Cannabis in Canada](https://yomu.fyi/post/engineering-a-historic-moment-shopify-gets-ready-for-cannabis-in-canad.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: Jason Hiltz-Laforge
- Published: Feb 7, 2019
- Type: Problem & solution

Canada’s 2018 cannabis legalization required Shopify to support a new, tightly regulated retail industry while keeping personal information in Canada. Using Google Cloud Platform’s Montreal region, the team created a Canadian Shopify instance with Terraform-managed projects, regional clusters, segregated workloads, PCI-compliant payment infrastructure, and a regional data warehouse. Because consumer behavior and peak demand were uncertain, engineers modeled traffic scenarios, ran repeated load tests across storefront actions, tested bot attacks, held pre-mortems, and provisioned five times projected launch traffic and sales. Multiple safeguards—including application-level endpoint checks, firewalls, limited secrets, unique SSL certificates, monitoring, and alerting—were layered to prevent data from crossing jurisdictions. Merchants opened across time zones, and the first legal sale occurred on Shopify’s point of sale in Newfoundland at 12:01 NST, followed by launches with zero downtime, excellent performance, and secure Canadian data storage and transmission.


### [Running Apache Kafka on Kubernetes at Shopify](https://yomu.fyi/post/running-apache-kafka-on-kubernetes-at-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: Sam Obeid
- Published: Nov 7, 2018
- Type: Explainer

Shopify adopted Apache Kafka in 2014 as a data bus for reliable messaging, initially collecting events and aggregating logs, then expanded to multi-tenant regional and aggregate clusters delivering billions of messages daily. In 2017, it migrated Kafka infrastructure from data centers to the cloud with zero downtime by mirroring regional data to both aggregate clusters before moving producers and consumers to cloud clusters. Shopify chose Kubernetes over GCP virtual machines because StatefulSets provide ordered, graceful deployment and scaling, while Custom Resources and Controllers support cluster-specific automation. The implementation uses self-built Kafka images in an own registry, persistent volumes, node and inter-pod affinity controls, taints, rack-awareness across zones, readiness probes, and resource-monitoring sidecars to limit restart and availability risks. The migration completed successfully, and Shopify now runs multiple regional clusters plus an aggregate cluster that mirrors data for the data warehouse.


### [Iterating Towards a More Scalable Ingress](https://yomu.fyi/post/iterating-towards-a-more-scalable-ingress.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: Francisco Mejia
- Published: Aug 31, 2018
- Type: Benchmark

Shopify’s infrastructure uses Kubernetes to serve millions of requests per minute, and its growth exposed ingress performance problems during frequent application deployments. After replacing Google Cloud Load Balancer Controller (glbc) with endpoint-aware ingress-nginx, the Edgescale team focused on reducing the NGINX reloads that caused latency increases. They added dynamic configuration: the controller POSTs endpoint lists to an NGINX HTTP endpoint, where lua-nginx-module updates an in-memory shared dictionary instead of regenerating the configuration file and reloading NGINX. Reload avoidance preserves keepalive connections and load-balancing state while reducing the resource effects of replacing worker processes. Tests found ingress-nginx outperformed glbc by multiple orders of magnitude at the 99.99th latency percentile or higher, while dynamic configuration produced its largest gains during deploys, especially beyond the 80th percentile.


### [Shopify’s Infrastructure Collaboration with Google](https://yomu.fyi/post/shopify-s-infrastructure-collaboration-with-google.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: Dale Neufeld
- Published: Mar 26, 2018
- Type: Announcement

Shopify announces that it is building its Cloud with Google after more than a decade of operating the foundation of its commerce cloud through its own data centers and incremental public-cloud adoption. Its infrastructure evolved from database and application sharding, performance testing, automated failovers, and Docker-based immutable infrastructure toward Kubernetes orchestration, after the company initially built minimal container-management features itself. During the migration, Shopify created Shop Mover, a selective database migration tool that rebalances shops between shards with an average of 2.5s downtime per shop, and moved more than 50% of data-center workloads to Google Cloud. It also grew to more than 400 production services, consolidated them on Kubernetes through a platform-as-a-service, and contributed to and used Grafeas for software supply-chain auditing and governance.


### [Dissecting Kubernetes Deployments](https://yomu.fyi/post/dissecting-kubernetes-deployments.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Damien Mathieu
- Published: Feb 22, 2018
- Type: Explainer

Kubernetes Deployments provide declarative updates for Pods and ReplicaSets, enabling application versions to roll out gradually rather than restarting every Pod at once. The Deployment controller uses informers and triggers for Deployment, ReplicaSet, and Pod events, then synchronizes state by creating ReplicaSets and scaling old and new sets according to replica counts, max surge, and available Pods. During a rolling update, it waits for new Pods to respond before terminating old ones; progress deadlines help detect stalled rollouts, including crash loops. After completion, old ReplicaSets are cleaned up but a configured number is retained for rollback, which requires changing image tags rather than continually using :latest. The post also shows a Go informer that watches Deployment creation and update events and explains that the trigger-based design supports custom controllers.


### [How Shopify Governs Containers at Scale with Grafeas and Kritis](https://yomu.fyi/post/how-shopify-governs-containers-at-scale-with-grafeas-and-kritis.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: Jonathan jonathan.pulsifer@shopify.com
- Published: Oct 12, 2017
- Type: Explainer

Shopify describes using Grafeas and Kritis to govern container software across a platform handling more than 6,000 container builds per day and 330,000 images in its primary registry. Grafeas serves as a central store for metadata about container origins, builds, base images, layers, registry pulls, installed packages, vulnerabilities, and deployment status, supporting auditing and lifecycle strategies. During CI, audits produce attestations, which Shopify signs with PGP; Kritis uses named attestation authorities and Kubernetes policies to require specific attestations before deployment. Policies can require containers to be built by Shopify, sourced from a trusted repository, free of newly introduced vulnerabilities, tested, non-root, and deployed with the appropriate security context. Shopify also generates attestations from kubeaudit results for workload manifests, helping catch runtime-security regressions before deployment.


[Newer posts](https://yomu.fyi/topic/kubernetes/page/7.md)