---
title: "Kubernetes"
description: "150 posts about Kubernetes, summarised, each linking to the original."
---

# Kubernetes
> 150 posts about Kubernetes, summarised, each linking to the original.

## Articles

### [IBM and UC Berkeley Diagnose Why Enterprise Agents Fail Using IT-Bench and MAST](https://yomu.fyi/post/ibm-and-uc-berkeley-diagnose-why-enterprise-agents-fail-using-it-bench.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Ayhan Sebin, Rohan Arora, Saurabh Jha
- Published: Feb 18, 2026
- Type: Benchmark

IBM Research and UC Berkeley analyze how agentic LLM systems fail in enterprise IT automation, including incident triage, logs and metrics queries, and Kubernetes actions across long-horizon tool loops. To move beyond benchmark success rates, the analysis applies MAST (Multi-Agent System Failure Taxonomy) to execution traces, converting failures into structured signatures across system design, inter-agent misalignment, and task verification. It covers 310 ITBench SRE traces from Gemini-3-Flash, Kimi-K2, and GPT-OSS-120B, whose mean recalls were 75.5%, 28.6%, and 12.4%, respectively. Gemini averaged 2.6 failure modes per failed trace, compared with 4.7 for Kimi and 5.3 for GPT-OSS-120B, while FM-3.3, Incorrect Verification, was identified as the strongest predictor across models; the analysis recommends external verification, explicit termination controls, and clarification branches.


### [Balancing cost and reliability for Spark on Kubernetes](https://yomu.fyi/post/balancing-cost-and-reliability-for-spark-on-kubernetes.md)
- Company: [Notion](https://yomu.fyi/company/notion.md)
- Author: Justin Lee
- Published: Feb 12, 2026
- Type: Problem & solution

Notion describes how it reduced Spark compute costs on Kubernetes while maintaining reliability across production workloads. After moving from fixed EC2 clusters on Amazon EMR to EMR on EKS, it used Karpenter, EKS Auto Mode, and the MostAllocated scheduler for dynamic provisioning and bin-packing. Spot Instances then exposed a failure mode: one interruption could remove many bin-packed executors, causing shuffle loss and expensive recomputation. Spot Balancer addresses this by labeling executor creation through a Kubernetes webhook and maintaining each job’s configured spot-to-on-demand ratio, with per-job counts stored in Redis. Named stability weights make the cost-reliability trade-off explicit, and reported savings reached 60–90 percent while jobs that previously failed from interruptions finished reliably.


### [SkyPilot at Shopify: Multi-cloud GPUs without the pain](https://yomu.fyi/post/skypilot-at-shopify-multi-cloud-gpus-without-the-pain.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jan 26, 2026
- Type: Explainer

Shopify uses SkyPilot to run machine-learning training workloads across GPU clusters on Nebius and GCP, addressing scarce hardware, fragmented capacity, and provider-specific interfaces. Persistent Kubernetes clusters remain managed by Shopify, while SkyPilot launches jobs and a company plugin routes requests, validates labels, injects configuration, and applies organizational policies. H200 workloads go to Nebius for InfiniBand-equipped distributed training, while L4 and CPU-only jobs go to GCP; labels also connect jobs to cost ownership, Kueue quotas, priority classes, and development-environment rules. The plugin automatically configures InfiniBand access, shared caches, and temporary storage, reducing manual setup and speeding repeated workloads. The resulting YAML-based interface preserves direct resource control while hiding cloud-selection decisions, giving teams common workflows as providers, prices, and GPU availability change.


### [Docker lazy loading at Grab: Accelerating container startup times](https://yomu.fyi/post/docker-lazy-loading-at-grab-accelerating-container-startup-times.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Huong Vuong
- Published: Jan 21, 2026
- Type: Problem & solution

Large Docker images for Airflow and Spark Connect caused minute-scale downloads, delaying cold starts and weakening autoscaling responsiveness. The post describes testing eStargz and Seekable OCI (SOCI) with containerd remote snapshotters, which expose registry-backed layers through a virtual lowerdir and fetch file content on demand. In production, SOCI delivered 30–40% faster startup times for Airflow and Spark Connect, while benchmark results showed SOCI matching standard images’ 5.0-second Airflow application startup; eStargz took 25.0 seconds. Configuration tuning increased download concurrency, unpack concurrency, and chunk size, reducing fresh-node image download time from 60 seconds to 24, a 60% improvement. Grab chose SOCI because Bottlerocket supports it natively, while noting that lazy loading redistributes rather than removes download work and requires workload-specific tuning.


### [Grab's service mesh evolution: From Consul to Istio](https://yomu.fyi/post/grab-s-service-mesh-evolution-from-consul-to-istio.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Hilman Kurniawan
- Published: Jul 16, 2025
- Type: Problem & solution

Grab describes migrating its service mesh from Consul and the Catcher fallback across a microservices ecosystem of more than 1,000 services spanning VMs, EKS clusters, AWS, and GCP. The existing setup exposed fleet-wide risk from a single Consul server issue, added fallback complexity, and lacked advanced traffic management, security controls, and robust multi-cluster support. After evaluating Application Load Balancer, Istio, AWS Lattice, and Linkerd for stability, performance, integration, and operational simplicity, the team selected Istio for its Kubernetes integration, flexible deployment models, and feature set. The design uses multiple control planes on dedicated Kubernetes clusters, active-active pairs, organization-based mesh boundaries, custom resource definitions, and cross-cluster endpoint discovery to support isolation, availability, routing, security, and observability. Migration began in Q4 2024 with GCP-to-AWS work alongside gRPC and HTTP traffic initiatives, using gradual shifting, rollback mechanisms, and performance monitoring while teams manage legacy systems and cross-mesh connectivity.


### [OpenTelemetry, Kubernetes, and Fir: Putting it All Together](https://yomu.fyi/post/opentelemetry-kubernetes-and-fir-putting-it-all-together.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Ken W. Alger
- Published: Jul 9, 2025
- Type: Explainer

Modern cloud-native applications span distributed microservices and dynamic environments, making performance diagnosis and failure analysis dependent on observability. The post explains how Fir, Heroku’s next-generation platform, combines Kubernetes infrastructure with OpenTelemetry to provide standardized visibility while hiding Kubernetes complexity behind Heroku’s interface. Fir runs on AWS EKS, uses OCI images and Cloud Native Buildpacks, and includes a Heroku OpenTelemetry Collector that gathers telemetry from applications, the Heroku router, and the Heroku API. OpenTelemetry’s APIs, SDKs, Collector, and OTLP support logs, metrics, and traces, with drains routing data to dashboards, CLI logs, or external observability platforms. The text also distinguishes OTLP gRPC and HTTP transports, noting gRPC efficiency and features versus HTTP’s firewall compatibility, and presents Fir’s open standards as a way to support scaling, hybrid environments, and reduce lock-in.


### [Evolution of Product Classification at Shopify: From Categories to Comprehensive Product Understanding](https://yomu.fyi/post/evolution-of-product-classification-at-shopify-from-categories-to-comp.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: May 8, 2025
- Type: Explainer

Shopify’s product classification system evolved from a 2018 logistic-regression model using TF-IDF into a product-understanding platform designed to capture categories, attributes, characteristics, metadata, and content-safety signals. A 2020 multimodal image-and-text approach improved classification, but growing product complexity led Shopify to combine Vision Language Models with its Standard Product Taxonomy, spanning more than 10,000 categories and over 1,000 attributes across more than 26 business verticals. The production pipeline uses two-stage category and attribute prediction, with category context shaping the second call, and runs on Kubernetes with NVIDIA GPUs; FP8 quantization, Nvidia Dynamo in-flight batching, and KV caching improve inference efficiency. A multi-LLM annotation system independently evaluates products and uses arbitration when models disagree, helping build training data for consistent predictions. The resulting system processes over 30 million predictions daily, delivers an 85% predicted-category acceptance rate, and doubles hierarchical precision and recall versus the earlier neural network approach.


### [Tinder’s migration to Elasticsearch 8](https://yomu.fyi/post/tinder-s-migration-to-elasticsearch-8.md)
- Company: [Tinder](https://yomu.fyi/company/tinder.md)
- Author: Tinder
- Published: Apr 1, 2025
- Type: Problem & solution

Tinder modernized its recommendation system by migrating its core Elasticsearch infrastructure from an end-of-life Elasticsearch 6 deployment to Elasticsearch 8 on Kubernetes. The legacy setup relied on manual EC2 provisioning and custom scripts, preventing the engineering team from utilizing modern vector search capabilities while incurring substantial operational overhead. Tinder implemented Elastic Cloud on Kubernetes integrated with its in-house Infrastructure as Code framework, Scaffold, while refactoring custom Java ranking plugins to handle Elasticsearch 8 breaking changes like prohibited negative scoring. The migration was validated through offline benchmarking, online verification, and a three-wave A/B testing rollout. This transition completed with zero outages and under 0.2% data discrepancy, reducing p99 search latency by 12% to 56% and cutting annual infrastructure costs by over one million dollars.


### [How Tinder Eased Development Pain With Ignis](https://yomu.fyi/post/how-tinder-eased-development-pain-with-ignis.md)
- Company: [Tinder](https://yomu.fyi/company/tinder.md)
- Author: Tinder
- Published: Nov 15, 2024
- Type: Problem & solution

Managing hundreds of microservices with disparate configurations forced Tinder engineers to deploy code to staging environments to verify every change, slowing development velocity. To shorten feedback loops, the team created Ignis, a Go-based CLI tool that lets engineers run an isolated service locally while proxying ingress and egress traffic through a multi-tenant development Kubernetes cluster. Ignis pairs with an API gateway for client testing and uses a custom Smart Router proxy to dynamically intercept downstream gRPC and HTTP requests via Kubernetes informers and header propagation. Since launching in 2019, the tool enabled Tinder to eliminate a dozen staging environments, decrease cloud spend, and accelerate onboarding, followed by a V2 release adding remote IDE support and monitoring.


### [Evolution of Catwalk: Model serving platform at Grab](https://yomu.fyi/post/evolution-of-catwalk-model-serving-platform-at-grab.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Vishal Sharma
- Published: Oct 1, 2024
- Type: Problem & solution

Grab developed and scaled Catwalk, an internal machine learning model serving platform, to address operational bottlenecks, low resource utilization, and deployment friction between data scientists and backend engineers. The platform transitioned from an admin-managed TensorFlow Serving setup into a low-code self-service system supporting PyTorch and ONNX, before replacing complex Helm charts with Kubernetes Custom Resource Definitions for declarative, blue-green deployment orchestration. To support complex business workflows and multi-model applications, Grab subsequently introduced Catwalk Orchestrator with bundled deployments that allow individual services to scale independently. Across two years, the orchestrator architecture expanded to 200 deployed applications serving approximately 1,400 production machine learning models.


### [Chimera Sandbox: A scalable experimentation and development platform for Notebook services](https://yomu.fyi/post/chimera-sandbox-a-scalable-experimentation-and-development-platform-fo.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Arkar Min Aung
- Published: Aug 27, 2024
- Type: Announcement

Grab's Chimera team built Chimera Sandbox to address resource accessibility challenges and enable rapid machine learning experimentation across technical and non-technical teams. The platform runs on multi-cluster Kubernetes infrastructure capable of scaling to thousands of nodes per cluster during peak loads. Users can provision customizable Docker environments with varying CPU and GPU configurations, collaborate in real time, and query data using Spark-on-Kubernetes integrations and helper magic cells. The workspace integrates with Grab's AI Gateway for LLM access alongside tools like MLflow, Optuna, RStudio, and LabelStudio on shared storage, backed by user-level cost attribution.


### [Fly Kubernetes does more now](https://yomu.fyi/post/fly-kubernetes-does-more-now.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Mar 7, 2024
- Type: Announcement

Fly Kubernetes is moving from early access to a closed beta as a managed Kubernetes service backed by Fly.io infrastructure. It provides the Kubernetes API while mapping core primitives to Fly resources: containers become Firecracker microVMs through flyd and Firecracker, pods become Fly Machines, networking uses an internal WireGuard mesh, services use the Fly Proxy, and storage uses Fly Volumes. Virtual Kubelet, running alongside K3s, receives pod requests and deploys them through the Fly Machines API, removing user-managed node capacity planning and hardware management. The beta supports workloads, including GPUs, across available regions, with Deployments/ReplicaSets, Services, DNS, and ephemeral and persistent volumes, but only part of the Kubernetes API. Missing capabilities include multi-container pods, StatefulSets, network policies, horizontal pod autoscaling, and emptyDir volumes; pricing is free during beta and planned at $75 per cluster monthly afterward, excluding resource costs.


### [Rethinking Stream Processing: Data Exploration](https://yomu.fyi/post/rethinking-stream-processing-data-exploration.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Shi Kai Ng
- Published: Jan 31, 2024
- Type: Problem & solution

Online data materialised in Kafka topics at Grab traditionally remained uninspected until ingested into the Data Lake as offline data, introducing latency and making stream processing applications difficult to develop. To enable earlier data exploration, Grab adopted Apache Zeppelin notebooks integrated with an internal data streaming platform. Zeppelin uses a Flink interpreter to convert interactive SQL queries into Flink jobs submitted to a Flink session cluster. The platform dynamically translates Protobuf schemas into SQL Data Definition Language statements to simplify table discovery. Security and compliance risks are addressed by combining mutual TLS authentication with Strimzi's Open Policy Agent to enforce read-only access policies and audit logging on Kafka topics.


### [Kafka on Kubernetes: Reloaded for fault tolerance](https://yomu.fyi/post/kafka-on-kubernetes-reloaded-for-fault-tolerance.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Fabrice Harbulot
- Published: Dec 26, 2023
- Type: Problem & solution

Grab's real-time data streaming platform, Coban, operates Kafka on AWS Elastic Kubernetes Service using Strimzi, allocating an entire EC2 worker node with NVMe instance store volumes to each broker. An initial architecture suffered from client connection errors, broken Network Load Balancer target groups, and zombie Persistent Volume Claims when worker nodes terminated. To achieve automated fault tolerance, the team integrated the AWS Node Termination Handler in Queue Processor mode with Auto Scaling lifecycle hooks, ensuring Kafka receives a SIGTERM to migrate partition leadership gracefully before shutdown. They also introduced the Kubernetes Cluster Autoscaler to dynamically provision replacement nodes during maintenance events and used the AWS Load Balancer Controller with TargetGroupBinding custom resources to dynamically update load balancer targets using IP mode.


### [Introducing Fly Kubernetes](https://yomu.fyi/post/introducing-fly-kubernetes.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Dec 18, 2023
- Type: Announcement

Fly.io introduces Fly Kubernetes (FKS), an implementation of Kubernetes that runs on its globally distributed container platform for users who want Kubernetes tooling and declarative orchestration. Fly.io says its earlier Nomad-based approach struggled with global scheduling at more than 200,000 applications and with placing workloads in users’ requested regions, so it moved state to individual workers and exposed the Fly Machines API, whose placement requests fail quickly rather than retry elsewhere. FKS combines K3s with Virtual Kubelet, mapping pod lifecycle operations to Fly Machines API calls while running K3s and Virtual Kubelet on one Fly Machine. A generated kubeconfig enables kubectl access, and Pods run on separate Fly Machines as the platform scales the cluster. The design provides declarative replica, autoscaling, and volume definitions plus automatic recovery for failed Machines, while retaining Fly Machines’ trade-off of failing when requested placement is unavailable.


### [Performance bottlenecks of Go application on Kubernetes with non-integer (floating) CPU allocation](https://yomu.fyi/post/performance-bottlenecks-of-go-application-on-kubernetes-with-non-integ.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Shubham Badkur
- Published: May 23, 2023
- Type: Problem & solution

Grab's real-time stream processing platform encountered severe consumer lag and CPU throttling when running Go-based Kafka consumer pipelines on Kubernetes. The issue originated when the Vertical Pod Autoscaler (VPA) scaled pod CPU allocations down to floating-point values such as 1.94 cores. Because AUTO-GOMAXPROCS rounds non-integer CPU limits down to integers, Go runtime thread allocation dropped to 1 core, significantly throttling pipeline throughput despite available pod capacity. Setting a minimum floor of 2 cores instantly restored CPU utilization to 95% and cleared the message backlog. To prevent similar throttling, the team utilized integer CPU scaling recommendations available in VPA v0.13 on Kubernetes 1.25 and above.


### [Safer deployment of streaming applications](https://yomu.fyi/post/safer-deployment-of-streaming-applications.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Shi Kai Ng
- Published: May 2, 2023
- Type: Problem & solution

Stateful stream processing frameworks like Apache Flink present unique deployment challenges because conventional canary and blue-green strategies can cause data inaccuracies or state divergence. Grab's real-time data platform team encountered risks of state loss, manual rollback overhead, and absent health checks in their Kubernetes and Spinnaker deployment pipeline. To resolve these operational issues, the team redesigned the deployment workflow around automated Flink savepointing and programmatic health monitoring. The new pipeline halts existing applications after capturing state snapshots and Kafka offsets, monitors target deployments via API health probes, and executes automated rollbacks using versioned ConfigMaps and replica metadata annotations. This automated process ensures state consistency during upgrades and eliminates manual intervention during deployment failures.


### [Unlocking Real-time Predictions with Shopify's Machine Learning Platform](https://yomu.fyi/post/unlocking-real-time-predictions-with-shopify-s-machine-learning-platfo.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Mar 10, 2023
- Type: Explainer

Shopify’s Merlin machine learning platform was expanded to support online inference, allowing trained models to return predictions as input arrives rather than through periodic batch jobs. The design targets varied Shopify use cases and requirements, including support for TensorFlow, PyTorch and XGBoost, low latency, autoscaling, rolling deployments, observability and model updates. Each use case runs as a dedicated service on Google Kubernetes Engine, with a Python serving layer that loads models and artifacts, exposes an inference endpoint, and can use the Comet ML model registry and Pano feature store. Separate staging and production configurations let teams tune resources and test model versions before release; Shopify reports that online inference is providing the intended low latency, scalability and faster iteration, while ensemble inference, platform-based monitoring and continuous training remain planned areas.


### [Performance Testing At Scale—for BFCM and Beyond](https://yomu.fyi/post/performance-testing-at-scale-for-bfcm-and-beyond.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jan 27, 2023
- Type: Benchmark

Shopify treats Black Friday Cyber Monday (BFCM) as its most critical period, and uses full-scale BFCM Scale Testing to assess whether the platform can handle peak traffic. Preparation combines capacity planning, resiliency testing, single-application performance testing, and repeated exercises that scale core services to their BFCM profiles before generating load. The hybrid architecture centers on Shopify Core, a monolithic Ruby application supported by services for storefront rendering, analytics, fraud analysis, payments, webhooks, and more. Five load flows cover browsing and buying, admin activity, flash sales, Storefront APIs, and Hydrogen and Oxygen, with Cronograma layering requests gradually and providing an abort switch. The exercises surfaced connection-count pressures, a Kubernetes ingress-nginx endpoint limit, and uneven load-balancer distribution; mitigations included refactoring, scaling dependent services, increasing pod capacity, and dedicated nginx node pools.


### [From Farmer to Security Engineer: How Dev Degree Helped Me Find My Dream Job](https://yomu.fyi/post/from-farmer-to-security-engineer-how-dev-degree-helped-me-find-my-drea.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jan 26, 2023
- Type: Explainer

Cailyn Edwards recounts how she moved from leaving high school and running an organic farm to a career in infrastructure security through Shopify’s Dev Degree program. Before entering the program, she completed a horticulture co-diploma, finished high school and the required math courses, and pursued computer science while needing to pay her mortgage and maintain work-life balance. Dev Degree’s training path and rotating placements exposed her to computer science fundamentals, pair programming, Ruby on Rails on the orders team, infrastructure security, and network foundations, leading her to discover a strong interest in cyber security and cloud infrastructure. After graduating in September 2021, she joined Infrastructure Security full-time and later reports projects, internal security talks, a promotion to Senior Infrastructure Engineer, and Kubernetes contributions including a contributor award.


[Newer posts](https://yomu.fyi/topic/kubernetes/page/5.md) · [Older posts](https://yomu.fyi/topic/kubernetes/page/7.md)