# Kubernetes
> 23 posts about Kubernetes, summarised, each linking to the original.

## Articles

### [How Netflix Simplified Batch Compute with Kueue](https://yomu.fyi/post/how-netflix-simplified-batch-compute-with-kueue.md)
- Company: [Netflix](https://yomu.fyi/company/netflix.md)
- Author: Netflix Technology Blog
- Published: Jun 22, 2026

Netflix transitioned its managed batch compute infrastructure from a homegrown solution called Compute Managed Batch to Kueue on its Titus container platform. CMB previously relied on custom scheduling and admission-only fair sharing without preemption, making feature development cumbersome as the Kubernetes ecosystem evolved. To modernize the platform transparently, Netflix mapped internal tenants to Cohorts and leaf tenants to ClusterQueues and LocalQueues while routing jobs through a custom Kueue router. Kueue operates alongside existing Titus scheduling profiles rather than replacing the kube-scheduler, preserving cluster placement efficiency. The migration was completed in four weeks across millions of batch workloads, significantly increasing average resource utilization through preemption-based fair sharing.


### [Palana (Part 2): Architecting isolation, identity, and auditability for AI agents](https://yomu.fyi/post/palana-part-2-architecting-isolation-identity-and-auditability-for-ai.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Kevin Littlejohn
- Published: Jun 21, 2026

Grab's Palana platform provisions isolated, Kubernetes-native runtime environments for autonomous AI agents using dedicated per-agent namespaces and role-based access controls. The architecture separates network enforcement across layers, applying Layer 3 and Layer 4 containment with Cilium and NetworkPolicy alongside Layer 7 application filtering evaluated by Open Policy Agent. Agent interactions with large language models route through a LiteLLM proxy wrapper that retrieves credentials from HashiCorp Vault based on Kubernetes pod context rather than client headers. Secrets management is divided between directly readable agent paths and proxy-only placeholder paths that prevent raw tokens from residing in runtime filesystems. An automated reaper monitors multi-source activity signals to shut down idle compute resources while preserving persistent storage and configuration state.


### [Palana (Part 1): Why Grab built a secure platform for autonomous AI Agents](https://yomu.fyi/post/palana-part-1-why-grab-built-a-secure-platform-for-autonomous-ai-agent.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Kevin Littlejohn
- Published: Jun 19, 2026

Autonomous AI agents introduce significant operational and security risks when granted network access, persistent state, and credentials. To address these concerns without impeding developer productivity, Grab created Palana, an in-house Kubernetes-native execution substrate. The platform isolates each agent workload within its own namespace, pairing it with dedicated storage, network policies, and role-based access control. Network egress is funneled through an Envoy and Open Policy Agent proxy layer that audits requests and injects credentials from HashiCorp Vault using placeholder tokens, keeping raw secrets outside the agent runtime. This design allows Grab to securely host hundreds of long-running workflows, remote coding environments, and automation bots.


### [Sitar-agent: Building a reliable dynamic configuration sidecar at scale](https://yomu.fyi/post/sitar-agent-building-a-reliable-dynamic-configuration-sidecar-at-scale.md)
- Company: [Airbnb](https://yomu.fyi/company/airbnb.md)
- Author: Bo Teng
- Published: Jun 4, 2026

Airbnb modernized sitar-agent, a Kubernetes sidecar that delivers dynamic configuration updates to thousands of polyglot service instances without requiring redeployments. The architecture uses periodic AWS S3 snapshots to enable fast, decoupled pod startups and maintains local file-based storage for the main container to read configs with in-memory caching. Key design decisions include retaining an isolated sidecar over an in-process library and optimizing pull-based polling with server-side caching.


### [Enhancing Flink deployment with shadow testing](https://yomu.fyi/post/enhancing-flink-deployment-with-shadow-testing.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Tee Long Lang
- Published: May 7, 2026

Production deployments of Apache Flink applications at Grab previously risked up to ten minutes of downtime when new versions failed under live traffic or could not resume from existing checkpoints. To eliminate these deployment failures, Grab's Coban data streaming team introduced an optional shadow testing stage directly into their production Kubernetes deployment pipeline. The shadow pipeline runs new application releases in parallel with the active application for a default observation period of one hour before prompting promotion to main. Dynamic runtime configuration based on an injected environment variable redirects source consumer groups and isolates output to dedicated shadow sinks, metrics, and logs.


### [Docker lazy loading at Grab: Accelerating container startup times](https://yomu.fyi/post/docker-lazy-loading-at-grab-accelerating-container-startup-times.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Huong Vuong
- Published: Jan 21, 2026

Grab addressed slow cold starts and auto-scaling bottlenecks caused by large container images across data platforms like Airflow and Spark Connect. To eliminate the requirement of downloading complete images before launching, the team evaluated Docker lazy loading via remote snapshotters using eStargz and Seekable OCI (SOCI). Unlike eStargz, which modifies image layers and increased application startup delays during benchmarks, SOCI stores index metadata separately as OCI Artifacts without altering image digests. In production on Amazon EKS, tuning SOCI concurrency and chunk parameters reduced fresh-node image download times by 60%, ultimately delivering a 30% to 40% reduction in P95 startup times.


### [Machine-learning predictive autoscaling for Flink](https://yomu.fyi/post/machine-learning-predictive-autoscaling-for-flink.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Minh Nhat Nguyen
- Published: Oct 30, 2025

Managing resource allocations for expanding Apache Flink stream-processing workloads often leads to over-provisioning or unstable reactive scaling. Reactive autoscaling with Kubernetes Horizontal Pod Autoscaling triggers severe restart spikes because pipelines must reprocess backlogged records from checkpoints, inducing feedback loops and scaling fluctuations. To prevent these spirals, a predictive vertical autoscaling system uses upstream Kafka message throughput as an independent workload metric. The architecture forecasts future workload using time-series models and maps the projected throughput to required TaskManager CPU allocations through a trained regression model. A custom controller vertically scales TaskManager CPU resources before traffic changes arrive, avoiding restart-induced latency spikes.


### [Grab's service mesh evolution: From Consul to Istio](https://yomu.fyi/post/grab-s-service-mesh-evolution-from-consul-to-istio.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Hilman Kurniawan
- Published: Jul 16, 2025

Grab operated over 1,000 microservices across hybrid infrastructure using Consul alongside a fallback mechanism called Catcher. Single-point-of-failure vulnerabilities in Consul servers and limited support for multi-cluster operations prompted an evaluation of alternative mesh technologies, ultimately leading to the selection of Istio. Grab avoided the standard single-control-plane-per-cluster pattern by deploying multiple external control planes in dedicated Kubernetes clusters arranged in active-active pairs. Migration began in Q4 2024, shifting traffic across AWS and GCP while handling both HTTP and gRPC protocols with gradual traffic-shifting and rollback mechanisms.


### [Supercharging LLM application development with LLM-Kit](https://yomu.fyi/post/supercharging-llm-application-development-with-llm-kit.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Boon Zhan Chew
- Published: Nov 29, 2024

Grab developed LLM-Kit to resolve standardization, security, observability, and infrastructure provisioning bottlenecks across its generative AI development efforts. Submitting an initial application request automatically triggers project generation in GitLab, outputting a modular codebase along with Terraform definitions for Amazon EKS and ECR. The scaffolded architecture incorporates FastAPI, LangChain, OpenID Connect authentication helpers, and PGVector alongside HashiCorp Vault for credential management. Observability and quality tracking are built in through direct integrations with Datadog and LangSmith evaluations. By providing these standardized paved-road templates, the framework has onboarded hundreds of generative AI applications and saved teams an estimated 1.5 weeks of initial setup time.


### [Evolution of Catwalk: Model serving platform at Grab](https://yomu.fyi/post/evolution-of-catwalk-model-serving-platform-at-grab.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Vishal Sharma
- Published: Oct 1, 2024

Grab developed and scaled Catwalk, an internal machine learning model serving platform, to address operational bottlenecks, low resource utilization, and deployment friction between data scientists and backend engineers. The platform transitioned from an admin-managed TensorFlow Serving setup into a low-code self-service system supporting PyTorch and ONNX, before replacing complex Helm charts with Kubernetes Custom Resource Definitions for declarative, blue-green deployment orchestration. To support complex business workflows and multi-model applications, Grab subsequently introduced Catwalk Orchestrator with bundled deployments that allow individual services to scale independently. Across two years, the orchestrator architecture expanded to 200 deployed applications serving approximately 1,400 production machine learning models.


### [Chimera Sandbox: A scalable experimentation and development platform for Notebook services](https://yomu.fyi/post/chimera-sandbox-a-scalable-experimentation-and-development-platform-fo.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Arkar Min Aung
- Published: Aug 27, 2024

Grab's Chimera team built Chimera Sandbox to address resource accessibility challenges and enable rapid machine learning experimentation across technical and non-technical teams. The platform runs on multi-cluster Kubernetes infrastructure capable of scaling to thousands of nodes per cluster during peak loads. Users can provision customizable Docker environments with varying CPU and GPU configurations, collaborate in real time, and query data using Spark-on-Kubernetes integrations and helper magic cells. The workspace integrates with Grab's AI Gateway for LLM access alongside tools like MLflow, Optuna, RStudio, and LabelStudio on shared storage, backed by user-level cost attribution.


### [Rethinking Stream Processing: Data Exploration](https://yomu.fyi/post/rethinking-stream-processing-data-exploration.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Shi Kai Ng
- Published: Jan 31, 2024

Online data materialised in Kafka topics at Grab traditionally remained uninspected until ingested into the Data Lake as offline data, introducing latency and making stream processing applications difficult to develop. To enable earlier data exploration, Grab adopted Apache Zeppelin notebooks integrated with an internal data streaming platform. Zeppelin uses a Flink interpreter to convert interactive SQL queries into Flink jobs submitted to a Flink session cluster. The platform dynamically translates Protobuf schemas into SQL Data Definition Language statements to simplify table discovery. Security and compliance risks are addressed by combining mutual TLS authentication with Strimzi's Open Policy Agent to enforce read-only access policies and audit logging on Kafka topics.


### [Kafka on Kubernetes: Reloaded for fault tolerance](https://yomu.fyi/post/kafka-on-kubernetes-reloaded-for-fault-tolerance.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Fabrice Harbulot
- Published: Dec 26, 2023

Grab's real-time data streaming platform, Coban, operates Kafka on AWS Elastic Kubernetes Service using Strimzi, allocating an entire EC2 worker node with NVMe instance store volumes to each broker. An initial architecture suffered from client connection errors, broken Network Load Balancer target groups, and zombie Persistent Volume Claims when worker nodes terminated. To achieve automated fault tolerance, the team integrated the AWS Node Termination Handler in Queue Processor mode with Auto Scaling lifecycle hooks, ensuring Kafka receives a SIGTERM to migrate partition leadership gracefully before shutdown. They also introduced the Kubernetes Cluster Autoscaler to dynamically provision replacement nodes during maintenance events and used the AWS Load Balancer Controller with TargetGroupBinding custom resources to dynamically update load balancer targets using IP mode.


### [Performance bottlenecks of Go application on Kubernetes with non-integer (floating) CPU allocation](https://yomu.fyi/post/performance-bottlenecks-of-go-application-on-kubernetes-with-non-integ.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Shubham Badkur
- Published: May 23, 2023

Grab's real-time stream processing platform encountered severe consumer lag and CPU throttling when running Go-based Kafka consumer pipelines on Kubernetes. The issue originated when the Vertical Pod Autoscaler (VPA) scaled pod CPU allocations down to floating-point values such as 1.94 cores. Because AUTO-GOMAXPROCS rounds non-integer CPU limits down to integers, Go runtime thread allocation dropped to 1 core, significantly throttling pipeline throughput despite available pod capacity. Setting a minimum floor of 2 cores instantly restored CPU utilization to 95% and cleared the message backlog. To prevent similar throttling, the team utilized integer CPU scaling recommendations available in VPA v0.13 on Kubernetes 1.25 and above.


### [Safer deployment of streaming applications](https://yomu.fyi/post/safer-deployment-of-streaming-applications.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Shi Kai Ng
- Published: May 2, 2023

Stateful stream processing frameworks like Apache Flink present unique deployment challenges because conventional canary and blue-green strategies can cause data inaccuracies or state divergence. Grab's real-time data platform team encountered risks of state loss, manual rollback overhead, and absent health checks in their Kubernetes and Spinnaker deployment pipeline. To resolve these operational issues, the team redesigned the deployment workflow around automated Flink savepointing and programmatic health monitoring. The new pipeline halts existing applications after capturing state snapshots and Kafka offsets, monitors target deployments via API health probes, and executes automated rollbacks using versioned ConfigMaps and replica metadata annotations. This automated process ensures state consistency during upgrades and eliminates manual intervention during deployment failures.


### [Zero trust with Kafka](https://yomu.fyi/post/zero-trust-with-kafka.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Fabrice Harbulot
- Published: Dec 7, 2022

Grab's real-time data platform team transitioned their large-scale Kafka infrastructure from basic network access controls to a zero-trust architecture. The platform implements mutual Transport Layer Security (mTLS) for offline peer authentication and encryption, driven by HashiCorp Vault's PKI engine and Strimzi on Kubernetes. Policy-Based Access Control is enforced using dedicated Open Policy Agent deployments per cluster, backed by GitOps workflows where topic owners approve JSON authorization rules. To simplify client integration, the team enhanced their Go SDK to handle ephemeral in-memory certificates, automatic renewals, and configurable retries. While the security posture improved, the Java encryption and decryption overhead caused a drop in streaming throughput.


### [Automatic rule backtesting with large quantities of data](https://yomu.fyi/post/automatic-rule-backtesting-with-large-quantities-of-data.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Chao Wang
- Published: Sep 8, 2022

Evaluating new or modified risk rules previously required Grab analysts to run slow offline Presto queries, manually construct payloads, or run rules in shadow mode for days. To standardize and accelerate this workflow, Grab developed an automated backtesting system powered by an AWS EMR Spark pipeline. Historical events are continuously ingested via Kafka and a Kubernetes stream pipeline into S3 using Snappy-compressed Parquet. Users configure replay intervals and rule definitions directly in the rule engine UI, which triggers asynchronous Spark jobs through Amazon SQS and Lambda. This automated simulation replaces multi-week shadow mode runs and generates downloadable aggregation metrics on transactions, user counts, and treatment outcomes.


### [How Kafka Connect helps move data seamlessly](https://yomu.fyi/post/how-kafka-connect-helps-move-data-seamlessly.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Wenli Wan
- Published: Apr 6, 2022

Grab's real-time data platform team, Coban, implemented a managed Kafka Connect ecosystem on Kubernetes to streamline moving data in and out of Apache Kafka. To resolve dual-write consistency issues and capture pre- and post-change data, the team integrated Debezium connectors to capture MySQL binlog events and accommodate database DDL migrations. For disaster recovery and stream migrations, Coban deployed MirrorMaker2 connectors managed via Terraform to handle message mirroring and consumer offset translation across AWS regions. Additionally, they developed a custom converter utilizing Confluent Schema Registry to transform Protobuf-serialized Kafka records into JSON for ingestion into Azure Event Hubs. This architecture enabled zero-downtime cluster migrations and robust cross-region disaster recovery.


### [Optimally Scaling Kafka Consumer Applications](https://yomu.fyi/post/optimally-scaling-kafka-consumer-applications.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Shubham Badkur
- Published: Oct 13, 2020

Grab's Coban platform runs Golang-based stream processing pipelines on Kubernetes, servicing roughly 400 billion events weekly from Kafka. The initial Horizontal Pod Autoscaler setup caused resource waste and uneven load distribution across Kafka partitions during scale-in and scale-out events. To resolve this, Grab moved to a fixed pod count matching the topic's partition count and adopted Vertical Pod Autoscaling, reducing resource usage versus requests by approximately 45%. The team also introduced Kubernetes priority classes to segment latency-sensitive workloads onto On-Demand nodes and non-critical jobs onto Spot instances. Additionally, overprovisioning via low-priority placeholder pods managed by Cluster Proportional Autoscaler enabled rapid pod rescheduling and reduced deployment delays.


### [Securing and Managing Multi-cloud Presto Clusters with Grab’s DataGateway](https://yomu.fyi/post/securing-and-managing-multi-cloud-presto-clusters-with-grab-s-datagate.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Vinnson Lee
- Published: Aug 24, 2020

Grab's data engineering team needed to manage data access across growing Presto workloads spanning hundreds of users, thousands of tables, and multiple clouds without exposing infrastructure churn to clients. Out-of-the-box Presto lacked fine-grained schema- and table-level access controls, unified endpoint routing, and seamless cluster switching. To solve this, the team built DataGateway, a smart HTTP proxy and abstraction layer deployed as microservices on Kubernetes. The platform parses incoming SQL statements, verifies user access control lists against target schemas and tables, and dynamically routes queries to appropriate AWS or Azure Presto clusters without requiring endpoint updates. This architecture enabled zero-disruption cluster migrations across 40 clusters and unified access management across a multi-cloud data lake environment.


[Older posts](https://yomu.fyi/topic/kubernetes/page/2.md)