---
title: "Kubernetes"
description: "150 posts about Kubernetes, summarised, each linking to the original."
---

# Kubernetes
> 150 posts about Kubernetes, summarised, each linking to the original.

## Articles

### [Palana (Part 1): Why Grab built a secure platform for autonomous AI Agents](https://yomu.fyi/post/palana-part-1-why-grab-built-a-secure-platform-for-autonomous-ai-agent.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Kevin Littlejohn
- Published: Jun 19, 2026
- Type: Explainer

Palana is Grab’s Kubernetes-native platform for running autonomous and semi-autonomous agents with isolation, controlled access, and operational oversight. The post describes why laptop-based tools and simply running agents in containers do not adequately address persistent state, identity, credentials, network access, auditability, and emergency control. Its architecture gives each agent a namespace, service account, storage, network policy, and Vault scope, while Envoy, ext-authz, and OPA mediate and log HTTP(S) egress; proxy-only secrets keep underlying credentials out of agent processes. A portal and pcli manage lifecycle, and external kill switches and idle shutdown provide controls outside the agent. Palana is used for development environments, Slack automation, framework testing, and long-running internal workloads, with deeper architecture details reserved for Part 2.


### [The open ecosystem around ClickHouse](https://yomu.fyi/post/the-open-ecosystem-around-clickhouse.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 17, 2026
- Type: Explainer

ClickHouse’s open ecosystem spans clients, dashboards, ingestion pipelines, projects built on its engine, and agents that query it, with 2,600+ contributors, 240,000+ commits, and hundreds of millions of downloads cited. It describes open standards including OpenTelemetry for observability, Model Context Protocol (MCP) for AI tooling, and Iceberg and Delta Lake for lakehouse data, plus Iceberg REST catalogs, Apache Polaris, and Unity Catalog. The surrounding stack includes ClickStack, chDB, Langfuse, LibreChat, official language clients, a Kubernetes operator, Postgres bridges such as PeerDB and pg\_clickhouse, and integrations with Airflow, dbt, Grafana, Fivetran, and Sigma. Specific updates include full Iceberg v2 support, Iceberg writes in 25.8, Paimon in 25.10, chDB v4’s DataStore API, and a Python client v1.0 with native async and 4x faster cold starts. The post frames this openness as supporting flexible tooling and community-led growth, reporting more than 200 integrations while noting that the ecosystem’s next direction remains unknown.


### [Cisco Talos empowers threat researchers while reducing TCO by 75% with ClickHouse Cloud](https://yomu.fyi/post/cisco-talos-empowers-threat-researchers-while-reducing-tco-by-75-with.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 15, 2026
- Type: Problem & solution

Cisco Talos operates a threat-intelligence reputation service that classifies file hashes from Cisco product telemetry, enabling researchers to make security decisions about benign and potentially malicious artifacts. Previously, a custom pipeline moved data from Databricks Delta and S3 through EKS, where staging and aggregated tables supported an API; the self-managed design retained about a petabyte of compressed, replicated data on 32 high-compute nodes. The team rebuilt the system in four phases, using Terraform, SSO/SAML, sharded ClickPipes for a 12-month backfill, shadow traffic, and a validated cutover to ClickHouse Cloud on AWS. The resulting architecture uses EventBridge, SQS, ClickPipes, SharedMergeTree, object storage, and AWS PrivateLink, delivering less than a minute of queue lag, zero data loss, a 100% response match across more than 1,000 hashes, zero downtime, roughly 90% lower storage costs, and about 75% lower TCO while reclaiming over 300 engineer hours annually.


### [Ingesting the Milky Way: Petabyte-Scale with Zerobus Ingest](https://yomu.fyi/post/ingesting-the-milky-way-petabyte-scale-with-zerobus-ingest.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Aleksandar Tomić, Victoria Bukta, Nikola Obradović, Danilo Najkov, Branko Grbić, Milos Milovanovic
- Published: Jun 11, 2026
- Type: Benchmark

The post benchmarks Zerobus Ingest, a managed, serverless, push-based service that writes producer data directly to Delta tables governed by Unity Catalog, against a petabyte-scale telemetry workload. Using NASA’s NEOWISE dataset and Locust, the test modeled fan-in from 2,048 concurrent streams, using Protocol Buffer 2 data over approximately 24 hours. The design replaces static partition-based ordering with stream-connection ordering, allowing heuristic routing across pods, dynamic partitioning, and autoscaling while existing streams drain. It also uses zeroparser, a zero-copy protobuf decoder whose design relies on Rust’s lifetime system and supports dynamic descriptors at about 1 GB/s per CPU core. The test sustained 12 GB/s to one table, ingested 1.04 trillion rows, and reached 1 petabyte within 24 hours; Zerobus Ingest is generally available, with additional APIs on its roadmap.


### [AI Serving Platform That Adapts to Your Model](https://yomu.fyi/post/ai-serving-platform-that-adapts-to-your-model.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Anshul Gupta
- Published: Jun 10, 2026
- Type: Problem & solution

Databricks Custom Model Serving addresses the operational burden of serving custom models, whose resource profiles, traffic patterns, and latency requirements vary widely from small CPU classifiers to large GPU-backed language models. Its fully managed platform packages MLflow models and uses isolated Kubernetes deployments, model-appropriate runtimes, and a short request path to limit interference and per-request overhead. At the center, the AutoPilot Pod Autoscaler combines active concurrency and queue signals for horizontal scaling with CPU, GPU, and memory measurements for model-aware target-concurrency adjustment, allowing one controller to adapt across workloads. Warm pools, provisioned concurrency, and zero-downtime updates address startup and deployment concerns, while reported production results include 90%+ cost savings for some customers, up to 2x improvement in p99 and p50 latency, 100K+ QPS, and 99.99% availability.


### [Sitar-agent: Building a reliable dynamic configuration sidecar at scale](https://yomu.fyi/post/sitar-agent-building-a-reliable-dynamic-configuration-sidecar-at-scale.md)
- Company: [Airbnb](https://yomu.fyi/company/airbnb.md)
- Author: Bo Teng
- Published: Jun 4, 2026

Airbnb modernized sitar-agent, a Kubernetes sidecar that delivers dynamic configuration updates to thousands of polyglot service instances without requiring redeployments. The architecture uses periodic AWS S3 snapshots to enable fast, decoupled pod startups and maintains local file-based storage for the main container to read configs with in-memory caching. Key design decisions include retaining an isolated sidecar over an in-process library and optimizing pull-based polling with server-side caching.


### [Multigres v0.1 Alpha: an operating system for Postgres](https://yomu.fyi/post/multigres-v0-1-alpha-an-operating-system-for-postgres.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: sugu\_sougoumarane
- Published: Jun 4, 2026
- Type: Announcement

Multigres v0.1 alpha is an open-source release of a system intended to provide Vitess-grade horizontal scaling, high availability, and operational simplicity for Postgres. It manages Postgres instances with sharding, connection pooling, automatic failover, and backup orchestration, while this release adds advanced pooling, automatic failovers, and a Kubernetes operator. Its high-availability protocol builds on unmodified Postgres replication, supports user-defined durability policies, resolves split-brain scenarios without losing succeeded commits, and permits replica changes while running. The release is suitable for experimentation but not production workloads: v0.1 is single-shard, sharding is absent, the custom-resource API may change, backward compatibility is not guaranteed, and performance benchmarks are still in progress.


### [Coding Is No Longer the Constraint: Scaling Developer Experience to Teams and Agents at Spotify](https://yomu.fyi/post/coding-is-no-longer-the-constraint-scaling-developer-experience-to-tea.md)
- Company: [Spotify](https://yomu.fyi/company/spotify.md)
- Author: Spotify Engineering
- Published: Jun 3, 2026
- Type: Problem & solution

Spotify experienced rapid codebase growth alongside widespread adoption of AI coding tools, resulting in a 76% increase in pull request frequency. To manage fleet-wide codebase maintenance and complex refactoring, the company combined its Fleetshift automation platform with Honk, a background coding agent powered by Claude. Honk runs inside Kubernetes pods with access to CI build environments and integrates with Backstage via Model Context Protocol tools to inspect component metadata. Standardized service architectures and active lint guardrails enable the agent to maintain consistency and auto-correct invalid design patterns during execution. By automating routine migrations, Spotify reduced fleet-wide Java backend updates from multi-week cross-team efforts down to three days handled by a single engineer.


### [Reliable LLM Inference at Scale](https://yomu.fyi/post/reliable-llm-inference-at-scale.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Ying Chen, Wendy Hu, Ankit Mathur, Mike Eastham, Pei-Lun Liao, Wai Wu, Arjun DCunha
- Published: May 27, 2026
- Type: Explainer

Databricks describes an inference platform serving open-source and proprietary frontier models at more than 125T tokens per month, with reliability as the central challenge under spiky, multi-tenant demand. It identifies failures and latency risks from expensive GPU configurations, variable input and output lengths, long-context requests, multimodal preprocessing, and engine-level hangs. Its architecture combines rate limiting, capacity management, model units, Dicer load-aware sticky routing, and autoscaling; model units estimate multidimensional request cost and allocate capacity more predictably. For runtime reliability, prioritized black-box health checks trigger Kubernetes liveness restarts for silent hangs, completing detection and recovery in less than five minutes while eliminating false probe failures. Image-processing changes, including Torchvision processors and OMP\_NUM\_THREADS configuration, removed CPU throttling and increased completed requests per second by more than threefold with the same replicas and load; bursty workloads also achieved over 80% GPU savings versus static peak provisioning.


### [How the lakebase architecture stays resilient to cloud failures](https://yomu.fyi/post/how-the-lakebase-architecture-stays-resilient-to-cloud-failures.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Jasraj Dange, Hans Norheim
- Published: May 27, 2026
- Type: Explainer

Lakebase is adapting its reliability architecture to agentic and on-demand workloads, which create far more control-plane activity, database starts, and pressure on finite cloud capacity. Lakebase uses stateless Postgres compute over zone-resilient storage, dedicated multi-availability-zone computes for optional high availability, and a vertically autoscaling virtualization layer that reduces dependence on cloud provisioning control planes. A dedicated data-plane controller is being developed for hot-path start and suspend operations, while cell-based regional deployments isolate failures and allow capacity to expand incrementally. The reliability program also uses component and whole-availability-zone failure injection, consistency checks, Service Level Indicators, and Service Level Objectives, including startup, failover, storage, and API measurements. The stated target is for every database to exceed 99.99% monthly availability; reported 2026 attainment through April shows 99.75% to 99.85% of databases meeting that threshold.


### [Protecting your Supabase projects from npm supply chain attacks](https://yomu.fyi/post/protecting-your-supabase-projects-from-npm-supply-chain-attacks.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: katerina\_skroumpelou
- Published: May 26, 2026
- Type: Explainer

The post explains why npm supply-chain attacks matter to Supabase projects: Edge Functions, the Supabase CLI, and packages including supabase-js, @supabase/ssr, and @supabase/server all depend on npm. Attackers can compromise maintainers, register typosquats, or exploit build pipelines; once malicious code reaches node\_modules, lifecycle scripts may read environment variables, cloud metadata, tokens, kubeconfig, and SSH keys. The proposed defense is layered: upgrade to pnpm 11, increase minimumReleaseAge to several days, pin sensitive dependency versions, commit and review lockfiles, disable or allowlist install scripts, verify package names, and pin GitHub Actions to full commit SHAs. Supabase is publishing a canonical security guide, hardening its GitHub Actions, adding secret-handling warnings to API documentation, and coordinating communications; the post also recommends rotating credentials after a potentially compromised install and using scanners as supplementary defense.


### [Solving the Identity Crisis for AI Agents](https://yomu.fyi/post/solving-the-identity-crisis-for-ai-agents.md)
- Company: [Uber](https://yomu.fyi/company/uber.md)
- Published: May 21, 2026
- Type: Explainer

Uber describes its 2025 effort to give AI agents verifiable identities and preserve attribution as agents act for users across compositional, multi-step workflows. The problem is that conventional human and workload identities expose only generic service identities, dropping originating users and intermediate agents across hops and weakening audits, authorization, and incident response. The design extends Uber’s Zero Trust Architecture with an Agent Registry, an AI Agent Mesh, Security Token Service (STS), MCP Gateway, downstream systems, and an AI Gateway, while integrating identity and context propagation into the agent SDK. Workloads obtain cryptographically signed workload SVIDs from SPIRE, and STS issues short-lived, scoped JWTs for each hop; MCP Gateway authenticates and authorizes tool calls. Production metrics reported show P99 STS token-exchange latency consistently below 40 milliseconds, while the roadmap places dynamic access control and unified policy enforcement above the identity foundation.


### [Building Blocks for Foundation Model Training and Inference on AWS](https://yomu.fyi/post/building-blocks-for-foundation-model-training-and-inference-on-aws.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Keita Watanabe, Pavel Belevich, Aman Shanbhag
- Published: May 11, 2026
- Type: Explainer

The post explains how foundation-model scaling now spans pre-training, post-training, and test-time compute, creating shared infrastructure requirements rather than a single pre-training curve. It examines how AWS infrastructure and open-source tools support these workloads across accelerated compute, networking, storage, orchestration, ML frameworks, and observability. Technical detail includes NVIDIA H100, H200, B200, and B300 GPU characteristics, NVLink/NVSwitch scale-up communication, and EFA-based scale-out networking for distributed collectives. The architecture also incorporates EC2 P instances, tiered storage, Slurm, Kubernetes, SageMaker HyperPod, CUDA, NCCL, PyTorch, Prometheus, and Grafana. The conclusion presents integration points between these layers as the basis for diagnosing bottlenecks and making scaling decisions across the foundation-model lifecycle.


### [How Superhuman and Databricks built a 200K QPS inference platform together](https://yomu.fyi/post/how-superhuman-and-databricks-built-a-200k-qps-inference-platform-toge.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Myke Troianovskyi, Christoph Stüber, Wai Wu, Arjun DCunha, Amine El Helou, Tian Ouyang, Jarek Odzga, Alex Coleman
- Published: May 8, 2026
- Type: Problem & solution

Superhuman replaced a DIY vLLM-based serving stack for its grammar-correction model with Databricks model serving, targeting peak traffic above 200,000 QPS, sub-second P99 latency, four-nines reliability, and no quality regression. The platform combines an Endpoint Discovery Service with power-of-two-choices load balancing, concurrency-based autoscaling, and asymmetric scale policies to reduce hotspots and handle rapid traffic ramps. Lazy-loading container images changed startup from several minutes to a few seconds by fetching metadata first and retrieving 4MB blocks on demand. Runtime work on H100 GPUs raised per-pod throughput from 750 to 1,200 QPS: FP8 weight quantization contributed up to 30%, while a multiprocessing server added 20%, alongside C++ and asynchronous scheduling optimizations. The teams validated changes through shadow testing and evaluation harnesses, and Superhuman is migrating additional models and workflows to the platform.


### [Enhancing Flink deployment with shadow testing](https://yomu.fyi/post/enhancing-flink-deployment-with-shadow-testing.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Tee Long Lang
- Published: May 7, 2026
- Type: Problem & solution

Grab’s Coban data streaming team addresses production-only Apache Flink deployment failures, which can leave an application unavailable for about 10 minutes while the system observes it and rolls it back. Its Shadow Testing stage runs a new version in parallel with the current Main version, replaying production data without affecting Main outputs and allowing behavior and regressions to be assessed before promotion. The Shadow deployment uses a shadow Kubernetes namespace, a shadow-prefixed application name, isShadow=true, a distinct Job ID, isolated sinks, and connector-specific separation such as suffixed Kafka consumer groups and shifted Debezium Server ID ranges. By default, Shadow runs for one hour while its job-manager status is monitored; users then continue to Main deployment, which receives a 10-minute health check and automatic rollback if unhealthy. The framework isolates Kafka and S3 outputs, prefixes metrics and logs for observability, and plans more connector support.


### [10 trillion samples a day: Scaling beyond traditional monitoring infra at Databricks](https://yomu.fyi/post/10-trillion-samples-a-day-scaling-beyond-traditional-monitoring-infra.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: David Yuan, Yi Jin, Karan Bavishi, HC Zhu, Joey Beyda
- Published: May 5, 2026
- Type: Problem & solution

Databricks’ monitoring infrastructure now tracks 5 billion active timeseries in real time and ingests more than 10 trillion samples daily, exposing scalability, reliability, cost, and operability limits in its older stack. The replacement, Pantheon, is a fork of CNCF Thanos deployed across more than 160 instances in about 70 regions and three cloud providers; tiered storage, differentiated memory retention, isolated replicated Receive groups, multitenancy, and a custom control plane support automated scaling and recovery. Pantheon’s largest instance holds about 300 million in-memory timeseries and handles nearly 1,000 PromQL queries per second, while migration reduced annual cloud costs by millions and monitoring downtime by roughly five times. For high-cardinality troubleshooting, Hydra preserves raw metrics in Delta tables, exposes them through Grafana and SQL, and unifies metric semantics across aggregated and raw paths, with freshness improvements planned.


### [Stop guessing in production: Full fidelity tracing at scale with ClickHouse and Odigos](https://yomu.fyi/post/stop-guessing-in-production-full-fidelity-tracing-at-scale-with-clickh.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Will Searle
- Published: May 5, 2026
- Type: Problem & solution

Distributed tracing frequently suffers from visibility gaps, broken context across asynchronous boundaries like Kafka, and high operational overhead from manually configuring OpenTelemetry SDKs and collectors. To resolve these issues, Odigos provides zero-code instrumentation via eBPF probes that dynamically attach to running workloads without modifying application code. An intelligent control plane within Odigos dynamically adjusts trace sampling and attribute enrichment based on real-time latency and error rates. The resulting telemetry exports over OTLP into ClickStack, an all-in-one backend featuring an OpenTelemetry Collector, ClickHouse storage, and the HyperDX query interface. This unified pipeline preserves end-to-end trace continuity, captures deep application and database context, and enables fast SQL-based trace analysis.


### [Zero downtime Upgrade: Yelp’s Cassandra 4.x Upgrade Story](https://yomu.fyi/post/zero-downtime-upgrade-yelp-s-cassandra-4-x-upgrade-story.md)
- Company: [Yelp](https://yomu.fyi/company/yelp.md)
- Author: Mark Surnin and Muhammad Junaid Muzammil, Software Engineer
- Published: Apr 7, 2026
- Type: Problem & solution

Yelp's Database Reliability Engineering team upgraded over a thousand Cassandra nodes from version 3.11 to 4.1 without downtime. The upgrade delivered reliability, observability, and performance improvements, including Java 11 support, configurable guardrails, and fixed incremental repairs. To avoid the high costs, consistency compromises, and data streaming delays of standing up separate datacenters, the team selected an in-place rolling upgrade strategy. The rollout orchestrated staged node updates alongside compatible versions of the Stargate proxy and custom data pipeline connectors without requiring client code modifications. Although Stargate 2.x introduced regressions on range and multi-partition queries that required downgrading to 1.x, the finished migration achieved up to a 58% reduction in p99 latency across key clusters.


### [ClickHouse BYOC on Google Cloud  now Generally Available](https://yomu.fyi/post/clickhouse-byoc-on-google-cloud-now-generally-available.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Aashish Kohli, Yiyang Shao
- Published: Mar 31, 2026
- Type: Announcement

ClickHouse Bring Your Own Cloud is now generally available on Google Cloud across all public ClickHouse Cloud Google regions without requiring a waitlist. The architecture separates the ClickHouse-managed control plane from the customer data plane, which runs entirely within the user Google Cloud project on Google Kubernetes Engine and Google Cloud Storage. An outbound-only, encrypted Tailscale zero-trust tunnel restricts control plane communication strictly to orchestration traffic, ensuring query data never leaves the customer environment. Deployment involves a three-step Terraform setup that establishes least-privilege IAM roles, configures VPC networking, and automates cluster provisioning. ClickHouse manages database and infrastructure maintenance through ArgoCD, employing a make-before-break strategy to add updated replicas before decommissioning older nodes across customizable release channels.


### [Run cloud agents in your own infrastructure](https://yomu.fyi/post/run-cloud-agents-in-your-own-infrastructure.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Katia Bazzi
- Published: Mar 25, 2026
- Type: Announcement

Cursor has made self-hosted cloud agents generally available, allowing code, tool execution, and build artifacts to remain within an organization’s network. Each agent runs in an isolated virtual machine or dedicated worker with a terminal, browser, and full desktop; it can clone repositories, configure environments, write and test code, and push changes for review. A worker connects outbound to Cursor’s cloud over HTTPS, receives tool calls from the agent harness, and returns execution results, while organizations can run long-lived or single-use workers and manage large fleets through a Helm chart, Kubernetes operator, or fleet management API. The product retains cloud-agent capabilities including multi-model support, plugins, and team permissions, while giving teams access to internal caches, dependencies, endpoints, and security controls without maintaining a separate agent stack.


[Newer posts](https://yomu.fyi/topic/kubernetes/page/4.md) · [Older posts](https://yomu.fyi/topic/kubernetes/page/6.md)