---
title: "Kubernetes"
description: "150 posts about Kubernetes, summarised, each linking to the original."
---

# Kubernetes
> 150 posts about Kubernetes, summarised, each linking to the original.

## Articles

### [Zero trust with Kafka](https://yomu.fyi/post/zero-trust-with-kafka.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Fabrice Harbulot
- Published: Dec 7, 2022
- Type: Problem & solution

Grab's real-time data platform team transitioned their large-scale Kafka infrastructure from basic network access controls to a zero-trust architecture. The platform implements mutual Transport Layer Security (mTLS) for offline peer authentication and encryption, driven by HashiCorp Vault's PKI engine and Strimzi on Kubernetes. Policy-Based Access Control is enforced using dedicated Open Policy Agent deployments per cluster, backed by GitOps workflows where topic owners approve JSON authorization rules. To simplify client integration, the team enhanced their Go SDK to handle ephemeral in-memory certificates, automatic renewals, and configurable retries. While the security posture improved, the Java encryption and decryption overhead caused a drop in streaming throughput.


### [Automatic rule backtesting with large quantities of data](https://yomu.fyi/post/automatic-rule-backtesting-with-large-quantities-of-data.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Chao Wang
- Published: Sep 8, 2022
- Type: Problem & solution

Evaluating new or modified risk rules previously required Grab analysts to run slow offline Presto queries, manually construct payloads, or run rules in shadow mode for days. To standardize and accelerate this workflow, Grab developed an automated backtesting system powered by an AWS EMR Spark pipeline. Historical events are continuously ingested via Kafka and a Kubernetes stream pipeline into S3 using Snappy-compressed Parquet. Users configure replay intervals and rule definitions directly in the rule engine UI, which triggers asynchronous Spark jobs through Amazon SQS and Lambda. This automated simulation replaces multi-week shadow mode runs and generates downloadable aggregation metrics on transactions, user counts, and treatment outcomes.


### [Spin Cycle: Shopify’s SFN Team Overcomes a Cloud-Development Spiral](https://yomu.fyi/post/spin-cycle-shopify-s-sfn-team-overcomes-a-cloud-development-spiral.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 4, 2022
- Type: Problem & solution

Shopify’s Fulfillment Network (SFN) worked to run its monolithic Rails application in Spin, a cloud-based development tool that provides on-demand Kubernetes containers instead of local service environments. Because SFN glues together Shopify and many external services, the team had to decide whether to run, mock, or share each dependency, while frequent changes to SFN, Spin, and those dependencies created regressions. The three-engineer team identified a human feedback dynamic: painful Spin experiences discouraged adoption, which allowed more Spin-specific breakage, while regular use could expose regressions before merge. To break that spiral, the team prioritized basic functionality for common validation flows, fixed disruptions there, and asked developers to use Spin daily and add missing capabilities; the intended outcome was a self-reinforcing “Loop of Happy Equilibrium,” though the transition is described as a hypothesis.


### [Spin Infrastructure Adventures: Containers, Systemd, and CGroups](https://yomu.fyi/post/spin-infrastructure-adventures-containers-systemd-and-cgroups.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 15, 2022
- Type: Incident / postmortem

Spin’s infrastructure team investigated instance instability after Kubernetes nodes experienced frequent failures, including roughly five nodes per day, and instances showed memory use that could exceed their assigned limits. Spin runs instances as Kubernetes pods, with systemd inside each container managing initialization and processes, and the team compared resource behavior across kubectl exec and Spin shell sessions. Stress tests showed that processes started through kubectl exec entered the kubepods cgroup hierarchy, while processes launched through Spin shell entered a hierarchy with an effectively unlimited memory limit, causing usage to appear on the node rather than the pod. Experiments comparing containerd, Docker, and Podman identified systemd cgroup delegation as the relevant difference: Podman isolated systemd and its child processes correctly, while the production setup used cgroupfs. The team identified switching COS to the systemd cgroup driver as the forthcoming fix, while noting that the bind-mounted cgroup filesystem enabled the escape.


### [The Journey to Cloud Development: How Shopify Went All-in on Spin](https://yomu.fyi/post/the-journey-to-cloud-development-how-shopify-went-all-in-on-spin.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 9, 2022
- Type: Problem & solution

Shopify’s development environment, the Ruby-based dev tool and xhyve-backed railgun VM, no longer handled the growing number of repositories, services, integrations, and resource demands around its monoliths. Developers needed more services, faced sluggish code-build-test loops, and encountered costly state changes when tophatting collaborators’ work. The Environments team concluded that development environments needed cloud-like elasticity, then tested an automatically configured local Kubernetes cluster and easy creation of Google Compute Engine VMs; the source also describes Isospin and Spin as the resulting direction. Local Kubernetes did not gain broad organic traction, while the current Isospin implementation is described as the team’s best iteration for developers working on majestic monoliths. The team plans to evolve it into an application-development library built on dev, add dependency inference, and address the tension between persistent environments and Kubernetes pod relocation.


### [Lessons Learned From Running Apache Airflow at Scale](https://yomu.fyi/post/lessons-learned-from-running-apache-airflow-at-scale.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: May 23, 2022
- Type: Explainer

Shopify describes operating Apache Airflow 2.2 on Kubernetes with the Celery executor and MySQL 8 for data extraction, machine-learning training, Iceberg maintenance, and DBT modeling. In its largest environment, Shopify runs over 10,000 DAGs, averages over 400 tasks running at a given moment, and executes over 150,000 runs per day, while facing slow file access, metadata growth, ownership tracking, DAG-author control, traffic bursts, and resource contention. The authors describe replacing per-pod GCSFuse access with an in-cluster NFS volume synchronized to GCS, retaining GCS as the user-facing source of truth; they also use 28-day metadata retention, a YAML manifest, DAG policies, and scheduling controls including pools, priority weights, and Celery queues. These measures are presented as ways to improve file performance, limit database growth, identify owners, enforce constraints, and reduce workload interference, though the retention policy does not support durable history-dependent features such as long-running backfills.


### [Möbius: Shopify’s Unified Edge](https://yomu.fyi/post/mo-bius-shopify-s-unified-edge.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Apr 20, 2022
- Type: Problem & solution

Shopify had more than six traffic paths for its core, services, PII-restricted systems, and mTLS-authenticated APIs, making platform-wide features harder to deploy and traffic behavior harder to observe. Möbius and Shopify Front End (SFE) were designed together: Möbius terminates TCP and TLS near clients and forwards requests to SFE, while SFE routes them from regional clusters to application clusters. SFE uses nginx deployments with custom Lua and a centralized API fed by Kubernetes controllers, enabling routing and active/active or active/passive load balancing without DNS TTL delays. Möbius automates edge domain configuration, DNS ownership validation, and certificate acquisition through vendor APIs, so application owners generally only need to change DNS. The unified design standardizes edge protection and observability, supports future edge capabilities, and leaves exceptions for services whose dependencies could create crash loops.


### [How Kafka Connect helps move data seamlessly](https://yomu.fyi/post/how-kafka-connect-helps-move-data-seamlessly.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Wenli Wan
- Published: Apr 6, 2022
- Type: Problem & solution

Grab's real-time data platform team, Coban, implemented a managed Kafka Connect ecosystem on Kubernetes to streamline moving data in and out of Apache Kafka. To resolve dual-write consistency issues and capture pre- and post-change data, the team integrated Debezium connectors to capture MySQL binlog events and accommodate database DDL migrations. For disaster recovery and stream migrations, Coban deployed MirrorMaker2 connectors managed via Terraform to handle message mirroring and consumer offset translation across AWS regions. Additionally, they developed a custom converter utilizing Confluent Schema Registry to transform Protobuf-serialized Kafka records into JSON for ingestion into Azure Event Hubs. This architecture enabled zero-downtime cluster migrations and robust cross-region disaster recovery.


### [The Magic of Merlin: Shopify's New Machine Learning Platform](https://yomu.fyi/post/the-magic-of-merlin-shopify-s-new-machine-learning-platform.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Apr 6, 2022
- Type: Explainer

Shopify’s machine learning platform team redesigned its infrastructure to accommodate varied use cases, dependencies, data types, integrations, and production requirements. Merlin uses an open-source stack in which on-demand Merlin Workspaces run as short-lived Ray clusters on Kubernetes, while projects isolate code and dependencies in Docker-based environments. Its first iteration focuses on training and batch inference, with users moving from local or Jupyter-based prototypes to automated production workflows through the Merlin API, Airflow, or Oozie. Shopify validated the platform by migrating its product categorization workflows, using Ray Train for distributed TensorFlow training and Ray ActorPool for batch inference; planned extensions include online inference, model lifecycle management, and machine learning monitoring.


### [7 Tips For Optimizing Apache Flink Applications](https://yomu.fyi/post/7-tips-for-optimizing-apache-flink-applications.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Mar 8, 2022
- Type: Tutorial

Shopify presents lessons from operating large, stateful Apache Flink applications on Kubernetes with Google Kubernetes Engine, RocksDB, and checkpoints and savepoints stored in Google Cloud Storage. The guidance begins with profiling tools, including async-profiler, VisualVM, jemalloc with jeprof, and Eclipse Memory Analyzer, to investigate CPU, heap, native-memory, and leak-related problems. Disabling Kryo fallback exposed unsupported serializers such as Scala BigDecimal and Scala algebraic data types; replacing them with supported alternatives increased throughput by 20 percent. The post also recommends separate configuration profiles for high-throughput backfills and low-latency steady state, while describing a RocksDB native-memory failure in which disabling the block cache stopped repeated out-of-memory crashes without affecting application performance.


### [That Old Certificate Expired and Started an Outage. This is What Happened Next](https://yomu.fyi/post/that-old-certificate-expired-and-started-an-outage-this-is-what-happen.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jan 12, 2022
- Type: Incident / postmortem

On September 30th, 2021, Shopify nearly suffered a platform-wide outage after Let’s Encrypt’s old root certificate expired, exposing stale certificate data in internal clients rather than at the public edge. At 10:15 a.m., monitoring detected certificate errors after deployment-related connection changes, and investigation traced them to Ruby services that depended directly or indirectly on httpclient, whose embedded cacert.pem was six years old and did not trust the applicable certificate chain. Shopify forked the library, replaced cacert.pem with a current root certificate bundle, deployed it to canaries, and automated pull requests for affected repositories. The temporary fix restored the monolith, while systems were added to track root-certificate changes and update the fork automatically. Deployment locking kept most monolith pods from cycling, limiting the incident to canaries and causing secondary effects including delayed order confirmations, throttled marketing campaigns, and missing product recommendations.


### [Debugging Systems in the Cloud: MySQL, Kubernetes, and Cgroups](https://yomu.fyi/post/debugging-systems-in-the-cloud-mysql-kubernetes-and-cgroups.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 21, 2021
- Type: Incident / postmortem

KateSQL, Shopify’s Database-as-a-Service platform on GKE, investigated why otherwise comparable MySQL Pods took between 10 and 30 minutes to be created during instance replacement. Measurements found extreme differences for Pods with identical resources and datasets: one required 2,120 seconds for initialization and 1,104 seconds for startup, while another took 74 and 17 seconds, respectively; affected Pods also accumulated more slow queries involving temporary memory tables. Profiling showed InnoDB’s single-threaded buffer pool initialization spending substantial time in mmap calls and the Linux memory cgroup function mem\_cgroup\_commit\_charge, while affected nodes had roughly 50,000 memory cgroups. The team linked the slowdown to a Linux kernel cgroup memory leak and slab reparenting behavior, then improved performance through node replacement, memory.force\_empty testing, and GKE 1.18 upgrades with relevant fixes.


### [Shopify-Made Patterns in Our Rails Apps](https://yomu.fyi/post/shopify-made-patterns-in-our-rails-apps.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 13, 2021
- Type: Explainer

Shopify describes patterns and infrastructure adaptations used to keep its large Rails monolith maintainable as its organization and platform scale. Its dev command-line tool standardizes project setup, dependencies, services, environment variables, integrations, and command aliases across hundreds of active projects. Shopify Core uses pods to partition shops across independent databases and extends the shard concept across provisioning, deployment, load balancers, caching, and servers, while splitting code into domains without separate databases. The post also details safeguards including short-term Active Record migrations consolidated periodically into structure.sql, automated tests, a pod-aware maintenance task system, and a translation platform that commits translated strings through Git. Finally, Monorail imposes versioned JSON event schemas between Kafka producers and consumers and supports review and automated scrubbing of PII fields.


### [Shopify's Path to a Faster Trino Query Execution: Infrastructure](https://yomu.fyi/post/shopify-s-path-to-a-faster-trino-query-execution-infrastructure.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 9, 2021
- Type: Problem & solution

Shopify’s Trino-based interactive query infrastructure faced inconsistent latency and operational failures as workloads grew to 15 Gbps and more than 300 million rows per second. The team analyzed query volumes, resource-heavy datasets, CPU wall time, and failure scenarios, then used a Trino Query Replicator to reproduce historical degradation and isolate issues involving compressed Kafka JSON, shared cluster classes, resource allocation, JVM settings, and dataset statistics. They added workload-specific and ephemeral clusters, reduced concurrent queries, applied recommended JVM recompilation settings, and limited task drivers to 16; they also scaled worker pods to 61 cores and 220 GB memory, although scaling alone was unstable. The infrastructure changes produced a stable environment and reduced P95 execution time to below 30 seconds, moving more queries into the zero-to-five-second range, while the five-second target remained a future goal.


### [How to Reliably Scale Your Data Platform for High Volumes](https://yomu.fyi/post/how-to-reliably-scale-your-data-platform-for-high-volumes.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 8, 2020
- Type: Problem & solution

Shopify’s data platform prepared for BFCM 2020, when merchant sales exceeded $5.1 billion and platform throughput rose by an average of 150 percent. The Data Platform Engineering team focused on throughput risk, which grows with volume and is greatest near ingestion and processing, while prioritizing work through four service tiers. For Tier 1 and Tier 2 systems, engineers defined each service’s primary objective, identified adjustable controls such as job frequency, overlap intervals, CPU and memory requests, BigTable node pools, and ProxySQL connection queues, then used controlled load tests to expose capacity and recovery risks. The resulting work included cluster scaling, load-balancing adjustments, query and job tuning, and improved indices, schemas, dashboards, playbooks, SLOs, incident handling, and alerts. The post reports that BFCM 2020 completed without merchant-impacting incidents; only a handful of minor, self-healing incidents occurred, while monitoring and occasional tuning preserved spare capacity.


### [Capacity Planning at Scale](https://yomu.fyi/post/capacity-planning-at-scale.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 3, 2020
- Type: Explainer

Shopify’s BFCM preparation combines traffic forecasting, resource planning, and platform-wide scalability testing to reduce the risk of outages during its busiest sales period. The engineering team worked with data scientists to model expected traffic, reviewed historical numbers, added safety margins, and created a master Google Cloud resource plan covering CPUs, storage, machine types, locations, and project-level quotas. Because 2020 brought unusual uncertainty from COVID-19-driven changes in online commerce, Shopify modeled multiple demand scenarios and chose the more aggressive growth cases, while retaining buffers for changes and regional failover. It also ran the Oktoberfest scale-up on Google Kubernetes Engine, using internal load-testing tooling powered by go-lua and embedded Lua to simulate storefront browsing, cart additions, checkout, and mock payment processing. The exercise exposed bottlenecks, which teams fixed before repeating the test; the successful rerun increased confidence for BFCM and reinforced a longer-term aim of more automated, dynamic, and resilient systems.


### [Vouching for Docker Images](https://yomu.fyi/post/vouching-for-docker-images.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 1, 2020
- Type: Explainer

Kubernetes does not enable Docker image signature verification by default, leaving organizations to decide whether images are trusted before they run in a cluster. The post explains code signing through hashes and public/private keys, then describes Binary Authorization as a combination of metadata storage, an enforcer, and a signing service. Using Grafeas and Kritis, or GKE’s Binary Authorization, cluster policies require specified attestors to sign images; invalid or unsigned images are blocked before their containers start. Voucher supplies the missing signing component as a REST service running in Cloud Run or Kubernetes: build pipelines submit image paths, Voucher performs security checks, signs qualifying images, and records signatures in the metadata service. Its extensible checks can verify internal ownership, GitHub commits, approvals, builder identity, and vulnerability counts, while distinct cluster policies balance compliance requirements with faster experimentation; Voucher later moved into the Grafeas organization.


### [Optimally Scaling Kafka Consumer Applications](https://yomu.fyi/post/optimally-scaling-kafka-consumer-applications.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Shubham Badkur
- Published: Oct 13, 2020
- Type: Problem & solution

Grab's Coban platform runs Golang-based stream processing pipelines on Kubernetes, servicing roughly 400 billion events weekly from Kafka. The initial Horizontal Pod Autoscaler setup caused resource waste and uneven load distribution across Kafka partitions during scale-in and scale-out events. To resolve this, Grab moved to a fixed pod count matching the topic's partition count and adopted Vertical Pod Autoscaling, reducing resource usage versus requests by approximately 45%. The team also introduced Kubernetes priority classes to segment latency-sensitive workloads onto On-Demand nodes and non-critical jobs onto Spot instances. Additionally, overprovisioning via low-priority placeholder pods managed by Cluster Proportional Autoscaler enabled rapid pod rescheduling and reduced deployment delays.


### [Securing and Managing Multi-cloud Presto Clusters with Grab’s DataGateway](https://yomu.fyi/post/securing-and-managing-multi-cloud-presto-clusters-with-grab-s-datagate.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Vinnson Lee
- Published: Aug 24, 2020
- Type: Problem & solution

Grab's data engineering team needed to manage data access across growing Presto workloads spanning hundreds of users, thousands of tables, and multiple clouds without exposing infrastructure churn to clients. Out-of-the-box Presto lacked fine-grained schema- and table-level access controls, unified endpoint routing, and seamless cluster switching. To solve this, the team built DataGateway, a smart HTTP proxy and abstraction layer deployed as microservices on Kubernetes. The platform parses incoming SQL statements, verifies user access control lists against target schemas and tables, and dynamically routes queries to appropriate AWS or Azure Presto clusters without requiring endpoint updates. This architecture enabled zero-disruption cluster migrations across 40 clusters and unified access management across a multi-cloud data lake environment.


### [The Journey of Deploying Apache Airflow at Grab](https://yomu.fyi/post/the-journey-of-deploying-apache-airflow-at-grab.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Chandulal Kavar
- Published: Jul 14, 2020
- Type: Problem & solution

Engineering and data teams across Grab originally operated independent Apache Airflow instances, causing duplicate maintenance overhead and frequent job failures around scaling, logging, and dependency management. To resolve this, a dedicated team developed a centralized orchestration platform that runs isolated, containerized Airflow instances per team on Amazon EKS. The platform categorizes deployments into three size tiers and provisions dedicated Redis brokers, RDS metadata stores, and Vault secret sidecars using Terraform and custom Helm charts. Teams customize container images using shared GitLab CI/CD templates, while worker scaling is handled via Kubernetes Horizontal Pod Autoscalers. Today, the platform runs roughly 20 Airflow instances executing between 1,000 and 60,000 daily jobs per instance.


[Newer posts](https://yomu.fyi/topic/kubernetes/page/6.md) · [Older posts](https://yomu.fyi/topic/kubernetes/page/8.md)