Loading…
Topic
144 posts about Kubernetes. Every summary links to the original.
See the idea
Bare metal gives engineers direct control over GPU, network, and virtualization layers, but it shifts elasticity and lifecycle work onto the team. This trade-off map shows when that bargain pays off.
Argo CD is a GitOps controller that treats Git as the desired state and continuously reconciles it against what actually runs in each cluster. It maps the parts of that loop and why pull-based agents change its scale.
Binary Authorization blocks container images that lack the attestations a cluster policy requires. This comparison sets a strict blocking posture beside an audit-only one, and shows how to choose between them.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Shopify had more than six traffic paths for its core, services, PII-restricted systems, and mTLS-authenticated APIs, making platform-wide features harder to deploy and traffic behavior harder to observe. Möbius and Shopify Front End (SFE) were designed together: Möbius terminates TCP and TLS near clients and forwards requests to SFE, while SFE routes them from regional clusters to application clusters. SFE uses nginx deployments with custom Lua and a centralized API fed by Kubernetes controllers, enabling routing and active/active or active/passive load balancing without DNS TTL delays. Möbius automates edge domain configuration, DNS ownership validation, and certificate acquisition through vendor APIs, so application owners generally only need to change DNS. The unified design standardizes edge protection and observability, supports future edge capabilities, and leaves exceptions for services whose dependencies could create crash loops.
Grab's real-time data platform team, Coban, implemented a managed Kafka Connect ecosystem on Kubernetes to streamline moving data in and out of Apache Kafka. To resolve dual-write consistency issues and capture pre- and post-change data, the team integrated Debezium connectors to capture MySQL binlog events and accommodate database DDL migrations. For disaster recovery and stream migrations, Coban deployed MirrorMaker2 connectors managed via Terraform to handle message mirroring and consumer offset translation across AWS regions. Additionally, they developed a custom converter utilizing Confluent Schema Registry to transform Protobuf-serialized Kafka records into JSON for ingestion into Azure Event Hubs. This architecture enabled zero-downtime cluster migrations and robust cross-region disaster recovery.
Shopify’s machine learning platform team redesigned its infrastructure to accommodate varied use cases, dependencies, data types, integrations, and production requirements. Merlin uses an open-source stack in which on-demand Merlin Workspaces run as short-lived Ray clusters on Kubernetes, while projects isolate code and dependencies in Docker-based environments. Its first iteration focuses on training and batch inference, with users moving from local or Jupyter-based prototypes to automated production workflows through the Merlin API, Airflow, or Oozie. Shopify validated the platform by migrating its product categorization workflows, using Ray Train for distributed TensorFlow training and Ray ActorPool for batch inference; planned extensions include online inference, model lifecycle management, and machine learning monitoring.
Shopify presents lessons from operating large, stateful Apache Flink applications on Kubernetes with Google Kubernetes Engine, RocksDB, and checkpoints and savepoints stored in Google Cloud Storage. The guidance begins with profiling tools, including async-profiler, VisualVM, jemalloc with jeprof, and Eclipse Memory Analyzer, to investigate CPU, heap, native-memory, and leak-related problems. Disabling Kryo fallback exposed unsupported serializers such as Scala BigDecimal and Scala algebraic data types; replacing them with supported alternatives increased throughput by 20 percent. The post also recommends separate configuration profiles for high-throughput backfills and low-latency steady state, while describing a RocksDB native-memory failure in which disabling the block cache stopped repeated out-of-memory crashes without affecting application performance.
On September 30th, 2021, Shopify nearly suffered a platform-wide outage after Let’s Encrypt’s old root certificate expired, exposing stale certificate data in internal clients rather than at the public edge. At 10:15 a.m., monitoring detected certificate errors after deployment-related connection changes, and investigation traced them to Ruby services that depended directly or indirectly on httpclient, whose embedded cacert.pem was six years old and did not trust the applicable certificate chain. Shopify forked the library, replaced cacert.pem with a current root certificate bundle, deployed it to canaries, and automated pull requests for affected repositories. The temporary fix restored the monolith, while systems were added to track root-certificate changes and update the fork automatically. Deployment locking kept most monolith pods from cycling, limiting the incident to canaries and causing secondary effects including delayed order confirmations, throttled marketing campaigns, and missing product recommendations.
KateSQL, Shopify’s Database-as-a-Service platform on GKE, investigated why otherwise comparable MySQL Pods took between 10 and 30 minutes to be created during instance replacement. Measurements found extreme differences for Pods with identical resources and datasets: one required 2,120 seconds for initialization and 1,104 seconds for startup, while another took 74 and 17 seconds, respectively; affected Pods also accumulated more slow queries involving temporary memory tables. Profiling showed InnoDB’s single-threaded buffer pool initialization spending substantial time in mmap calls and the Linux memory cgroup function mem_cgroup_commit_charge, while affected nodes had roughly 50,000 memory cgroups. The team linked the slowdown to a Linux kernel cgroup memory leak and slab reparenting behavior, then improved performance through node replacement, memory.force_empty testing, and GKE 1.18 upgrades with relevant fixes.
Shopify describes patterns and infrastructure adaptations used to keep its large Rails monolith maintainable as its organization and platform scale. Its dev command-line tool standardizes project setup, dependencies, services, environment variables, integrations, and command aliases across hundreds of active projects. Shopify Core uses pods to partition shops across independent databases and extends the shard concept across provisioning, deployment, load balancers, caching, and servers, while splitting code into domains without separate databases. The post also details safeguards including short-term Active Record migrations consolidated periodically into structure.sql, automated tests, a pod-aware maintenance task system, and a translation platform that commits translated strings through Git. Finally, Monorail imposes versioned JSON event schemas between Kafka producers and consumers and supports review and automated scrubbing of PII fields.
Shopify’s Trino-based interactive query infrastructure faced inconsistent latency and operational failures as workloads grew to 15 Gbps and more than 300 million rows per second. The team analyzed query volumes, resource-heavy datasets, CPU wall time, and failure scenarios, then used a Trino Query Replicator to reproduce historical degradation and isolate issues involving compressed Kafka JSON, shared cluster classes, resource allocation, JVM settings, and dataset statistics. They added workload-specific and ephemeral clusters, reduced concurrent queries, applied recommended JVM recompilation settings, and limited task drivers to 16; they also scaled worker pods to 61 cores and 220 GB memory, although scaling alone was unstable. The infrastructure changes produced a stable environment and reduced P95 execution time to below 30 seconds, moving more queries into the zero-to-five-second range, while the five-second target remained a future goal.
Shopify’s data platform prepared for BFCM 2020, when merchant sales exceeded $5.1 billion and platform throughput rose by an average of 150 percent. The Data Platform Engineering team focused on throughput risk, which grows with volume and is greatest near ingestion and processing, while prioritizing work through four service tiers. For Tier 1 and Tier 2 systems, engineers defined each service’s primary objective, identified adjustable controls such as job frequency, overlap intervals, CPU and memory requests, BigTable node pools, and ProxySQL connection queues, then used controlled load tests to expose capacity and recovery risks. The resulting work included cluster scaling, load-balancing adjustments, query and job tuning, and improved indices, schemas, dashboards, playbooks, SLOs, incident handling, and alerts. The post reports that BFCM 2020 completed without merchant-impacting incidents; only a handful of minor, self-healing incidents occurred, while monitoring and occasional tuning preserved spare capacity.
Shopify’s BFCM preparation combines traffic forecasting, resource planning, and platform-wide scalability testing to reduce the risk of outages during its busiest sales period. The engineering team worked with data scientists to model expected traffic, reviewed historical numbers, added safety margins, and created a master Google Cloud resource plan covering CPUs, storage, machine types, locations, and project-level quotas. Because 2020 brought unusual uncertainty from COVID-19-driven changes in online commerce, Shopify modeled multiple demand scenarios and chose the more aggressive growth cases, while retaining buffers for changes and regional failover. It also ran the Oktoberfest scale-up on Google Kubernetes Engine, using internal load-testing tooling powered by go-lua and embedded Lua to simulate storefront browsing, cart additions, checkout, and mock payment processing. The exercise exposed bottlenecks, which teams fixed before repeating the test; the successful rerun increased confidence for BFCM and reinforced a longer-term aim of more automated, dynamic, and resilient systems.
Kubernetes does not enable Docker image signature verification by default, leaving organizations to decide whether images are trusted before they run in a cluster. The post explains code signing through hashes and public/private keys, then describes Binary Authorization as a combination of metadata storage, an enforcer, and a signing service. Using Grafeas and Kritis, or GKE’s Binary Authorization, cluster policies require specified attestors to sign images; invalid or unsigned images are blocked before their containers start. Voucher supplies the missing signing component as a REST service running in Cloud Run or Kubernetes: build pipelines submit image paths, Voucher performs security checks, signs qualifying images, and records signatures in the metadata service. Its extensible checks can verify internal ownership, GitHub commits, approvals, builder identity, and vulnerability counts, while distinct cluster policies balance compliance requirements with faster experimentation; Voucher later moved into the Grafeas organization.
Grab's Coban platform runs Golang-based stream processing pipelines on Kubernetes, servicing roughly 400 billion events weekly from Kafka. The initial Horizontal Pod Autoscaler setup caused resource waste and uneven load distribution across Kafka partitions during scale-in and scale-out events. To resolve this, Grab moved to a fixed pod count matching the topic's partition count and adopted Vertical Pod Autoscaling, reducing resource usage versus requests by approximately 45%. The team also introduced Kubernetes priority classes to segment latency-sensitive workloads onto On-Demand nodes and non-critical jobs onto Spot instances. Additionally, overprovisioning via low-priority placeholder pods managed by Cluster Proportional Autoscaler enabled rapid pod rescheduling and reduced deployment delays.
Grab's data engineering team needed to manage data access across growing Presto workloads spanning hundreds of users, thousands of tables, and multiple clouds without exposing infrastructure churn to clients. Out-of-the-box Presto lacked fine-grained schema- and table-level access controls, unified endpoint routing, and seamless cluster switching. To solve this, the team built DataGateway, a smart HTTP proxy and abstraction layer deployed as microservices on Kubernetes. The platform parses incoming SQL statements, verifies user access control lists against target schemas and tables, and dynamically routes queries to appropriate AWS or Azure Presto clusters without requiring endpoint updates. This architecture enabled zero-disruption cluster migrations across 40 clusters and unified access management across a multi-cloud data lake environment.
Engineering and data teams across Grab originally operated independent Apache Airflow instances, causing duplicate maintenance overhead and frequent job failures around scaling, logging, and dependency management. To resolve this, a dedicated team developed a centralized orchestration platform that runs isolated, containerized Airflow instances per team on Amazon EKS. The platform categorizes deployments into three size tiers and provisions dedicated Redis brokers, RDS metadata stores, and Vault secret sidecars using Terraform and custom Helm charts. Teams customize container images using shared GitLab CI/CD templates, while worker scaling is handled via Kubernetes Horizontal Pod Autoscalers. Today, the platform runs roughly 20 Airflow instances executing between 1,000 and 60,000 daily jobs per instance.
Grab's backend services process terabytes of data ingress per hour, generating recurring needs for stream transformations, joins, and time-windowed aggregations across diverse workloads. To support these asynchronous processing patterns across their Go ecosystem, the Coban team developed a managed, NoOps event sourcing and stream processing platform. The architecture packages stateless processing pipelines as Kubernetes deployments on AWS, polling Kafka event logs and using ScyllaDB as a shared metastore for stateful needs like deduplication and windowing. Stream processing pods combine ingestion triggers, a worker pool runtime, and user-provided domain logic plugins with customizable failure handling. This infrastructure scales to handle over 300 billion events weekly while maintaining workload isolation and elastic autoscaling.
Shopify manages a petabyte-scale fleet of MySQL servers across replica-sets, or shards, in three Google Cloud Platform regions, where file-based Xtrabackup archives made shard restores take more than six hours. To reduce that Recovery Time Objective, the team built Kubernetes CronJobs around the Compute Engine Persistent Disk snapshot API, scheduling snapshots every 15 minutes while accounting for regions, zones, instance roles, and MySQL consistency variables. Persistent Disk snapshots take about 20 minutes initially and typically under 10 minutes incrementally; restoring the latest snapshot to a new volume and starting MySQL, including InnoDB instance recovery and replication-lag recovery, typically brings RTO below 30 minutes. Retention tooling keeps the latest two snapshots plus policy-defined dailies and weeklies, while verification checks startup, GTID auto-positioning, and InnoDB page corruption; offsite tooling compresses, encrypts, and transfers exported data to another provider.
As machine learning adoption expanded at Grab, individual teams created fragmented model serving solutions that duplicated engineering effort and required data scientists to handle underlying infrastructure. To resolve these inefficiencies, Grab developed Catwalk, a self-service machine learning model serving platform. The system runs TensorFlow Serving containers across a managed Kubernetes cluster integrated with Grab's observability stack. Data scientists deploy or update models simply by saving files using the tf.saved_model API to dedicated Amazon S3 buckets, while Kubernetes automates orchestration, ingress routing, and pod autoscaling. Catwalk abstracts server management away from data scientists, shortens deployment timelines, and provides high availability during model version rollouts.
Shopify contributed a kubectl plugin to ingress-nginx, an open-source Kubernetes Ingress controller built on NGINX, because debugging and upgrading the heavily used controller often required long, frustrating sequences of kubectl commands. The plugin’s lint subcommand identifies ingress configuration that may need changes for a new release, while richer ingress listings expose individual paths, target services, and TLS configuration. It also wraps common kubectl operations, automatically selects an ingress pod, and helps inspect generated nginx.conf and dynamic configuration managed with openresty lua-nginx-module. The author reports substantially faster upgrades and debugging, comparing a rollout to version 0.24.1 with an earlier upgrade to 0.22.0 that took days, while acknowledging that growing Kubernetes expertise also contributed. Released with ingress-nginx 0.24.0, the plugin should also be compatible with 0.23.0.
Canada’s 2018 cannabis legalization required Shopify to support a new, tightly regulated retail industry while keeping personal information in Canada. Using Google Cloud Platform’s Montreal region, the team created a Canadian Shopify instance with Terraform-managed projects, regional clusters, segregated workloads, PCI-compliant payment infrastructure, and a regional data warehouse. Because consumer behavior and peak demand were uncertain, engineers modeled traffic scenarios, ran repeated load tests across storefront actions, tested bot attacks, held pre-mortems, and provisioned five times projected launch traffic and sales. Multiple safeguards—including application-level endpoint checks, firewalls, limited secrets, unique SSL certificates, monitoring, and alerting—were layered to prevent data from crossing jurisdictions. Merchants opened across time zones, and the first legal sale occurred on Shopify’s point of sale in Newfoundland at 12:01 NST, followed by launches with zero downtime, excellent performance, and secure Canadian data storage and transmission.
Shopify adopted Apache Kafka in 2014 as a data bus for reliable messaging, initially collecting events and aggregating logs, then expanded to multi-tenant regional and aggregate clusters delivering billions of messages daily. In 2017, it migrated Kafka infrastructure from data centers to the cloud with zero downtime by mirroring regional data to both aggregate clusters before moving producers and consumers to cloud clusters. Shopify chose Kubernetes over GCP virtual machines because StatefulSets provide ordered, graceful deployment and scaling, while Custom Resources and Controllers support cluster-specific automation. The implementation uses self-built Kafka images in an own registry, persistent volumes, node and inter-pod affinity controls, taints, rack-awareness across zones, readiness probes, and resource-monitoring sidecars to limit restart and availability risks. The migration completed successfully, and Shopify now runs multiple regional clusters plus an aggregate cluster that mirrors data for the data warehouse.
The best engineering reads, once a week.
