---
title: "Kubernetes"
description: "150 posts about Kubernetes, summarised, each linking to the original."
---

# Kubernetes
> 150 posts about Kubernetes, summarised, each linking to the original.

## Articles

### [Configure admission fair sharing in Red Hat build of Kueue 1.4](https://yomu.fyi/post/configure-admission-fair-sharing-in-red-hat-build-of-kueue-1-4.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Maysa De Macedo Souza
- Published: Sep 4, 2026
- Type: Tutorial

Red Hat build of Kueue 1.4 provides admission fair sharing for shared CPU, GPU, and high-performance computing clusters where FIFO admission can starve later workloads. The guide configures the feature through the Kueue custom resource, using either default values—30-minute usage half-life, five-minute sampling interval, and resource weights of 1—or custom settings, then creates a ResourceFlavor, a ClusterQueue using UsageBasedAdmissionFairSharing, and two LocalQueues. Kueue tracks each LocalQueue's historical consumption, adds a temporary penalty when a workload is admitted, and compares usage adjusted by queue weight to choose the lowest-use pending workload when quota returns. In the example, lq-a receives the first job, lq-b is admitted after job1 is deleted, and usage decays or grows on sampling ticks; usage ordering takes precedence over priority classes, which only break ties.


### [How to Carry User Identity Across Federated Kubernetes and AI Platforms](https://yomu.fyi/post/how-to-carry-user-identity-across-federated-kubernetes-and-ai-platform.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Elizabeth Goodman
- Published: Sep 3, 2026
- Type: Explainer

Federated data and AI platforms let users move among portals, datasets, notebooks, and assistants across clusters, but single sign-on alone does not carry trusted identity across control-plane and data-plane boundaries. The post presents a central identity gateway that owns platform sessions through OIDC, a shared session store, stateless data-plane gateways, and an identity-validation API. Regional gateways validate sessions through /gateway/userinfo, enforce local policy, and inject standardized identity headers instead of forwarding raw tokens or independently managing refresh and logout. This separates session ownership from request enforcement, supports consistent logout, and lets assistants pass delegated user identity to backend tools without broad service credentials. At NVIDIA, the approach reduced repeated login events by 55% across internal developer platforms spanning Kubernetes clusters in AWS and OCI, while establishing a foundation for unified platform shells and AI assistants.


### [Extend Layer 2 networks into Red Hat OpenShift Virtualization with BGP and EVPN](https://yomu.fyi/post/extend-layer-2-networks-into-red-hat-openshift-virtualization-with-bgp.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Miguel Duarte de Mora Barroso, Valentino Uberti
- Published: Sep 3, 2026
- Type: Tutorial

The article describes how to extend Layer 2 networks from a legacy data-center fabric into Red Hat OpenShift Virtualization using BGP and EVPN. It uses a three-node OpenShift lab with secondary NICs on a provider network, static IPv4 addressing, a VTEP CR for 172.19.0.0/24, FRRouting peering, and a Layer 2 primary CUDN with VNI 100. The configuration preserves imported VMs’ IP and MAC identities while maintaining isolation for tenant networks that may have overlapping address spaces. In the example, an external FRR router learns the VM MAC through EVPN and reaches 22.100.0.3; after the VM moves from node 3 to node 1, the MAC remains unchanged and the advertised VTEP changes from 172.19.0.102 to 172.19.0.100. The stated result is phased migration with Layer 2 adjacency to legacy application tiers and end-to-end tenant segmentation.


### [Hybrid cloud orchestration: Modernizing on-premises infrastructure management with AWS](https://yomu.fyi/post/hybrid-cloud-orchestration-modernizing-on-premises-infrastructure-mana.md)
- Company: [AWS](https://yomu.fyi/company/aws.md)
- Author: Sandeep Singh
- Published: Sep 1, 2026
- Type: Explainer

Managing distributed on-premises infrastructure across hundreds of sites creates inconsistency, manual lifecycle bottlenecks, fragmented visibility, and scalability problems, especially when hardware, operating system, Kubernetes, and application operations span thousands of servers. The proposed design centralizes orchestration in AWS while keeping execution on site, using hybrid connectivity, AWS Lambda, AWS Step Functions, Amazon EventBridge, DynamoDB, AWS Systems Manager, and supporting services. Redfish APIs provide vendor-agnostic bare-metal control for BIOS, firmware, power, and health operations, while EKS Anywhere deploys and operates management and workload clusters on local hardware; orders and inventory records track workflows and resource relationships. DynamoDB Streams also automate DNS, certificates, AWS access, and storage integrations, while centralized observability aggregates server, cluster, and application signals; the post presents architectural patterns and defers detailed implementation workflows and templates to later parts.


### [Orchestrate production RAG with OpenShift AI](https://yomu.fyi/post/orchestrate-production-rag-with-openshift-ai.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Ana Biazetti, Saad Zaher
- Published: Sep 1, 2026
- Type: Tutorial

This guide shows how to move a streaming retrieval-augmented generation (RAG) workflow from a monolithic Ray Data script to modular AI pipelines on OpenShift AI. It addresses failure recovery and run traceability by separating parsing and chunking, ingestion, model download, and deployment into five reusable components. Parsed and chunked JSONL is stored in S3-compatible storage so KFP can rerun ingestion without repeating document processing, and each run records parameters, timestamps, logs, and status. The data chain uses Docling’s HybridChunker and Ray map\_batches with ActorPoolStrategy, while the model chain downloads a Hugging Face model and deploys it through vLLM and KServe in parallel. For embedding ingestion, the workflow supports local CPU processing with granite-embedding-125m-english or a dedicated GPU vLLM service for larger collections and offers notebooks for deployment and query validation.


### [Rerouting the Stream: How Lyft Moved to the Apache Flink Operator](https://yomu.fyi/post/rerouting-the-stream-how-lyft-moved-to-the-apache-flink-operator.md)
- Company: [Lyft](https://yomu.fyi/company/lyft.md)
- Author: Maheep Myneni
- Published: Aug 31, 2026
- Type: Problem & solution

Lyft migrated its fleet of hundreds of critical streaming jobs from an internally developed Flink Kubernetes operator to the open-source Apache Flink Kubernetes operator. The legacy system imposed maintenance burden, feature gaps, outdated dependencies, fragile savepoint handling, and a rigid memory model that complicated varied workloads, including Apache Beam applications. Lyft used an incremental migration designed to preserve deployment patterns and team workflows while addressing CRD translation, state-machine behavior, and operational compatibility. The new platform added explicit last-state upgrades, autoscaling, automatic memory tuning, and safer upgrade mechanisms, though autotuning required trade-offs with in-place autoscaling and node capacity planning. After the multi-quarter transition, maintenance burden was minimized, the autoscaler right-sized a fleet previously overprovisioned by a few million dollars annually, and Lyft could contribute fixes upstream instead of maintaining its own operator.


### [Autoscaling Lakebase Postgres](https://yomu.fyi/post/autoscaling-lakebase-postgres.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Carlota Soto
- Published: Aug 31, 2026
- Type: Explainer

Lakebase Postgres addresses the waste and uncertainty of choosing a fixed database instance size before workload demand is known. Its decoupled architecture separates stateless compute from durable storage, allowing compute to start, stop, move, or resize without moving the database. The autoscaling algorithm combines CPU load, memory pressure, and a time-aware estimate of the Postgres working set, using modified HyperLogLog registers that store timestamps and estimate windows from one to 60 minutes. Scaling occurs live through autoscaler-agent, vm-monitor, Kubernetes scheduling, and NeonVM, with checks preventing unsafe downscales and scheduler approval for upscales. The system scales in both directions without dropping connections and operates across 100-millisecond, five-second, and 20-second timescales, including more than 32,000 resizes per month.


### [AI-powered multicluster management: Querying fleet health with OpenShift Lightspeed and Red Hat Advanced Cluster Management](https://yomu.fyi/post/ai-powered-multicluster-management-querying-fleet-health-with-openshif.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Diego Alvarez Ponce, Luiz Bernardo Levenhagen
- Published: Aug 31, 2026
- Type: Tutorial

Managing modern large-scale Kubernetes fleets across distributed clusters can make troubleshooting slow because administrators must build manual search queries and sift scattered outputs. The post presents a Technology Preview MCP server for Red Hat Advanced Cluster Management that translates natural-language prompts into structured searches across managed clusters, deployed with Helm in the hub cluster. OpenShift Lightspeed is configured through its OLSConfig custom resource to call the MCP endpoint, use a Kubernetes service-account token for internal authorization, and optionally skip approval for read-only searches. The examples cover finding clusters below OpenShift 4.21, locating pods in a specified cluster and namespace, and identifying an unhealthy VM across the fleet; one result finds dev-vm in an ErrorUnschedulable condition and provides diagnostic guidance.


### [Stop wasting GPU allocation in Kubernetes with GPU-pruner](https://yomu.fyi/post/stop-wasting-gpu-allocation-in-kubernetes-with-gpu-pruner.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Fahim Uddin
- Published: Aug 31, 2026
- Type: Problem & solution

GPU-pruner addresses idle GPU allocation in Kubernetes by identifying workloads whose hardware activity remains below a configured threshold, rather than relying on browser sessions, network traffic, or pod lifetime. It queries Prometheus for NVIDIA Data Center GPU Manager metrics, including engine activity and GPU utilization, and treats peak performance below 1% throughout a default 35-minute observation window as idle. After tracing an idle pod through ownerReferences or KServe labels, it scales the parent resource to zero, preserving workload metadata for later restoration. Slack notifications, grace periods, lookback checks, and dry-run mode provide safeguards, while Prometheus label handling must preserve the workload namespace and pod identity in DCGM telemetry.


### [Self-service backup for VMs and containers on OpenShift: No cluster-admin required](https://yomu.fyi/post/self-service-backup-for-vms-and-containers-on-openshift-no-cluster-adm.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Michal Pryc, Shubham Dilip Pampattiwar
- Published: Aug 28, 2026
- Type: Problem & solution

In multi-tenant Red Hat OpenShift clusters, Velero Backup and Restore resources live in the OADP installation namespace, leaving non-admin application teams dependent on platform staff. OADP self-service addresses this with namespace-scoped NonAdminBackup and NonAdminRestore resources that proxy Velero resources, while a controller validates requests against DataProtectionApplication policies and synchronizes phase, progress, and queue status back to the tenant namespace. Administrators can enforce retention, snapshot and file-system backup settings, resource inclusion or exclusion, and approval for tenant-provided backup storage; cluster-scoped resources such as SCCs, ClusterRoles, CRDs, and KubeVirt cluster preferences are automatically excluded. OADP 1.6 supports the same workflow for containers, stateful applications, and OpenShift Virtualization VMs, but self-service has no native schedules, cross-namespace restores, or short-lived credentials for tenant-provided storage locations.


### [Automating Red Hat OpenShift AI installations with Helm and GitOps](https://yomu.fyi/post/automating-red-hat-openshift-ai-installations-with-helm-and-gitops.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Davide Bianchi, Andrea Tarocchi
- Published: Aug 26, 2026
- Type: Tutorial

Deploying Red Hat OpenShift AI manually across multiple environments is error-prone due to the numerous required dependencies, including certificate managers, job queuing systems, and GPU operators. To streamline this lifecycle, the odh-gitops repository introduces the rhai-on-openshift-chart Helm chart. The chart automates operator provisioning through Operator Lifecycle Manager and configures custom resources such as DataScienceCluster. Dependency resolution is controlled via a tri-state model that automatically installs prerequisites like cert-manager and Red Hat Connectivity Link when dependent components are enabled. Using this chart, teams can deploy full platform installations or minimal inference stacks through standard Helm CLI commands or Argo CD GitOps pipelines.


### [GRPO fine-tuning on Red Hat OpenShift AI: Reinforcement learning from verifiable rewards with Training Hub](https://yomu.fyi/post/grpo-fine-tuning-on-red-hat-openshift-ai-reinforcement-learning-from-v.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Fiona Waters
- Published: Aug 26, 2026
- Type: Tutorial

Supervised fine-tuning for structured reasoning tasks like tool calling often demands expensive, labor-intensive dataset curation. Group Relative Policy Optimization (GRPO) resolves this by using programmatic reward functions to score multiple candidate responses without requiring human labels or a dedicated reward model. On Red Hat OpenShift AI, developers can implement GRPO using Training Hub, Kubeflow Trainer, and the Kubeflow SDK to fine-tune models like Qwen3-4B. The setup utilizes vLLM for candidate generation alongside LoRA weight updates, sharing storage via ReadWriteMany persistent volume claims. In a practical tool-calling test, five training iterations raised task accuracy from 33% to 67%, teaching the model to emit parseable function invocations rather than fabricated conversational text.


### [Restore LLM Inference Capacity in Seconds with Shadow Engine Recovery in NVIDIA Dynamo](https://yomu.fyi/post/restore-llm-inference-capacity-in-seconds-with-shadow-engine-recovery.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Michelle Horton
- Published: Aug 25, 2026
- Type: Problem & solution

LLM engine process failures typically trigger cold restarts that reload weights into HBM, compile kernels, and recapture CUDA graphs, leaving surviving workers to absorb displaced traffic for minutes. Shadow engine recovery, a preview feature in NVIDIA Dynamo, keeps a fully initialized standby on the same GPUs as the active engine. GPU Memory Service (GMS) preserves one physical weight copy across engine processes, while the parked shadow precomputes its CUDA context, graphs, communicators, and mappings but defers KV cache materialization. In a GLM-5.2 test, the shadow resumed service in 7.3 seconds after a worker failure, compared with 283 seconds for a cold restart, reducing post-fault p50 TTFT from 23,815 ms to 1,311 ms and improving decode rate from 12 to 46 tok/s/user. The preview does not cover hardware, node, or multi-node failures and currently starts promoted shadows with empty KV caches.


### [Red Hat OpenShift autoscaling with Cluster Autoscaler](https://yomu.fyi/post/red-hat-openshift-autoscaling-with-cluster-autoscaler.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Ramon Gordillo Gutierrez, Jose Ortiz Padilla
- Published: Aug 25, 2026
- Type: Explainer

Red Hat OpenShift supports compute scaling using Cluster Autoscaler integrated with the OpenShift machine API. The mechanism relies on paired ClusterAutoscaler and MachineAutoscaler custom resources to manage cluster-wide limits and per-MachineSet replica boundaries. Scaling decisions evaluate pending pods against node allocatable capacity and hypothetical node capacity annotations while respecting expander strategies like Random, LeastWaste, or Priority. Cluster Autoscaler checks for unnecessary nodes every 10 seconds, scaling down when utilization falls below configured thresholds and workloads can safely migrate. To mitigate provisioning delays, clusters can adopt overprovisioning via low-priority pause containers or use APIs like ProvisioningRequest and CapacityBuffer for predictive scaling.


### [How Databricks Uses AI to Accelerate Incident Investigation](https://yomu.fyi/post/how-databricks-uses-ai-to-accelerate-incident-investigation.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Avijeet Gupta, Bhuban Seth, Kusum Madarasu
- Published: Aug 24, 2026
- Type: Explainer

Databricks describes AI SRE, an AI-powered debugging agent for operating hundreds of microservices across more than 1,500 Kubernetes clusters, 70+ regions, and three clouds. It addresses incident investigations by assembling context from platform health, service telemetry, changes, dependencies, and team-specific runbooks, with automatic triage followed by interactive, natural-language investigation. The platform uses layered primitives, purpose-built APIs, and a core orchestration engine; team-owned agentic runbooks encode domain procedures, while controlled access and evidence links support auditing. The post reports support for more than 150 teams, 250+ weekly active users, and over 2,000 daily investigations, with users saving several hours of debugging time.


### [NVIDIA BlueField-4 Powers New Scale-In Network Infrastructure for Agentic AI Factories](https://yomu.fyi/post/nvidia-bluefield-4-powers-new-scale-in-network-infrastructure-for-agen.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Michelle Horton
- Published: Aug 24, 2026
- Type: Explainer

Agentic AI factories connect diverse users, agents, applications, data sources, and storage systems to massively accelerated compute, making dedicated DPU processing essential for line-rate networking, storage, and security. Scale-In evolves north-south access into a coordinated infrastructure domain built with BlueField-4, NVIDIA DOCA, and Spectrum-X Ethernet. BlueField-4 combines programmable control-plane processing with inline data-path engines, a 64-core Grace CPU, LPDDR5X memory, PCIe Gen6 host connectivity, and an 800 Gb/s network interface. The architecture offloads security, tenant isolation, storage protocols, provisioning, and telemetry from host CPUs; the source states that Spectrum-X storage paths can provide up to 1.45x the throughput of off-the-shelf Ethernet, while independent control and observability support more consistent operations as AI infrastructure scales.


### [Run LoRA fine-tuning on Red Hat OpenShift AI with Ray](https://yomu.fyi/post/run-lora-fine-tuning-on-red-hat-openshift-ai-with-ray.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Fiona Waters
- Published: Aug 24, 2026
- Type: Tutorial

Red Hat OpenShift AI 3.5 integrates Ray cluster runtime images with preinstalled Training Hub packages to support distributed fine-tuning algorithms including LoRA, SFT, OSFT, and GRPO. Using the CodeFlare software development kit within a Jupyter workbench, developers can provision an elastic single-GPU Ray cluster attached to shared ReadWriteMany persistent storage containing model weights and training datasets. The tutorial demonstrates fine-tuning the Qwen2.5-1.5B-Instruct model on the public sql-create-context dataset to produce direct, bare SQL output without explanatory conversational text. After training completes, the workbench loads and merges the resulting LoRA adapter checkpoints with the base model to evaluate output format correctness against sample database schema questions. The RayCluster automatically tears down resources once the job concludes, while the shared pipeline architecture allows switching between supported fine-tuning algorithms by modifying dataset inputs and configuration parameters.


### [Ensuring reliable OpenTelemetry ingestion at scale](https://yomu.fyi/post/ensuring-reliable-opentelemetry-ingestion-at-scale.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Aug 24, 2026
- Type: Problem & solution

LogHouse’s OpenTelemetry pipeline ingests 50 million events per second and stores 177 PiB of uncompressed data, but its initial agent-to-gateway design could not reliably absorb ClickHouse backpressure. In-memory queues caused instability during short-lived spikes or database outages, while local collector write-ahead logs made gateways stateful, slowed recovery, delayed fresh telemetry behind FIFO backlogs, and took about four hours to drain 1 TiB. The team rejected adding Kafka because operating a new tier-zero service across cloud regions was not justified for this use case, then designed a custom architecture using blob storage, the OTel failover connector, overflow notifications, and a catchup collector. In staging, failover redirected 120,000 events per second to S3 without changing gateway resource usage, and recovery resumed ClickHouse writes while the catchup collector drained the queue; the design scaled from 10 million to 50 million events per second, with added regional setup and less precise recovery visibility.


### [Improved failure reports on Red Hat OpenShift with the event-driven diagnostic operator](https://yomu.fyi/post/improved-failure-reports-on-red-hat-openshift-with-the-event-driven-di.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Periyamaruthu Mohanraj
- Published: Aug 21, 2026
- Type: Problem & solution

During major infrastructure incidents, critical system logs and diagnostic evidence are frequently overwritten, lost, or erased by emergency recovery efforts before engineers can investigate. To solve this visibility gap in multi-cluster environments, an open-source event-driven diagnostic operator was developed as an automatic first information report collector. Deployed on a central hub cluster, the operator monitors Kubernetes warning events rather than relying on heavy polling. When an event matches a predefined regular expression pattern, the operator identifies the target spoke cluster, copies its kubeconfig secret, and spins up an independent must-gather job. Diagnostic data is written to off-site or shared storage with a configurable retention window, preserving forensic context without human intervention.


### [Shopify powers observability for global-scale commerce with ClickHouse](https://yomu.fyi/post/shopify-powers-observability-for-global-scale-commerce-with-clickhouse.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Aug 20, 2026
- Type: Problem & solution

Shopify rebuilt its observability infrastructure after metrics, logs, and traces became fragmented across vendors, with unpredictable costs and limited cross-signal analysis. It created Observe, a unified platform that runs metrics, logs, traces, profiles, and exceptions on self-hosted ClickHouse, treating them as structured, time-ordered, high-dimensional events and serving roughly 50 million events per second normally and 100 million at BFCM peak, or about 110 GB/s. Kafka buffers telemetry for large synchronous durable writes, while materialized views support flexible schemas, autocomplete, typed hot keys, and identifier-based correlation; the latter reportedly makes targeted searches 10x faster. Shopify reports a 16x query improvement out of the box, exceeding 30x at peak performance, with data queryable in under a minute, and says it would likely choose ClickHouse Cloud if starting again because managed operations could reduce the self-hosting burden.


[Newer posts](https://yomu.fyi/topic/kubernetes/page/2.md) · [Older posts](https://yomu.fyi/topic/kubernetes/page/4.md)