Loading…
Topic
150 posts about Kubernetes. Every summary links to the original.
See the idea
A model deployed on AKS is open by default. See how MaaS policies insert Authorino authentication and Limitador rate limits in front of vLLM, so bad keys and over-quota requests stop at the gate.
Bare metal gives engineers direct control over GPU, network, and virtualization layers, but it shifts elasticity and lifecycle work onto the team. This trade-off map shows when that bargain pays off.
Argo CD is a GitOps controller that treats Git as the desired state and continuously reconciles it against what actually runs in each cluster. It maps the parts of that loop and why pull-based agents change its scale.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
The post overviews recent memory management evolutions in Red Hat OpenShift Virtualization, spanning from the Linux kernel to the orchestration planes. Historically, the platform uses a 1:1 memory request mapping, but data shows most VMs use only about 20% of allocated RAM, creating a delta that scales infrastructure with workload definitions rather than demand. Node-level optimizations include free page reporting (FPR), which asynchronously recovers pages freed by the guest OS without forcing allocation like ballooning, and KSM zero pages for guest-agnostic deduplication of zero-filled memory. At the cluster level, Multi-Gen LRU integrated with control groups enables swapping to fast secondary NVMe storage, and a PSI-aware descheduler proactively migrates VMs to relieve memory pressure across nodes. OpenShift Virtualization 4.22 provides an autopilot feature that automates configuration for generally available memory optimizations, while developer preview capabilities require manual opt-in activation by default.
Red Hat OpenShift Dev Spaces can be hardened for enterprise use by centralizing SCM authentication with OAuth, deploying a private OpenVSX extension registry, and enabling IntelliJ IDEA in restricted networks. The SCM integration uses a dedicated OAuth application and the scm-oauth-config Secret in the operator namespace, while disabling the default CA bundle mount reduces etcd overhead but requires an explicit corporate CA ConfigMap with keys ca.crt and githost to prevent TLS failures. A private OpenVSX registry with PostgreSQL-only architecture replaces external extensions, and automated governance rejects prereleases, timestamp builds, oversized patches, and previews through version-aware dependency resolution and idempotent publication tracking. For JetBrains support in air-gapped clusters, administrators mirror IDE tarballs to an internal repository and set editorsDownloadUrls; developers connect via JetBrains Gateway using either dashboard- or gateway-initiated workflows, both relying on an existing oc login session. This configuration gives platform teams oversight of authentication, extensions, and IDE tooling.
AWS Resilience Hub’s Next generation, announced September 18, 2026, adds three capabilities to align workload scope and resilience requirements with running environments. Amazon EKS label support selects Kubernetes resources using existing labels with AND logic, establishing precise resource boundaries without reorganizing namespaces. Generative AI-powered Dependency Insights summarizes discovered dependencies and highlights patterns warranting resilience review, such as cross-Region dependencies questioning recovery design. Resilience policy sharing through AWS Organizations lets a central team define requirements once and make them available to member accounts, with policy events tracking adoption. The post walks through scoping a checkout service on EKS using labels, examining dependencies via synthetic traffic, and applying a shared tier-1-critical policy with a 99.9% availability SLO, 5-minute multi-AZ RTO, 1-minute RPO, 15-minute multi-Region RTO, and 60-minute backup intervals.
NVIDIA has released the complete Run:ai platform as open source software, expanding the NVIDIA DSX AI Factory Platform and giving customers, developers, and ecosystem partners production-grade AI workload and GPU orchestration capabilities. Run:ai has dynamically allocated GPU resources and managed workloads on Kubernetes across data processing, training, and inference. The release includes the full platform plus four independently adoptable components: KAI Scheduler, GPU Fractioning Operator, KAI Resource Management, and Workload Map. Teams can deploy the stack from the NVIDIA Run:ai repository in the DSX AI Factory GitHub organization or via an NGC Community Product Helm chart, without a paid subscription or enterprise support entitlement. Ecosystem partners including Crusoe, Mirantis, Nebius, OpenNebula Systems, Rafay, Saturn Cloud, Spectro Cloud, and vCluster are integrating the platform or its components into their offerings.
Red Hat AI Inference 3.5 extends its Model-as-a-Service (MaaS) governance layer to non-OpenShift Kubernetes distributions, including Azure Kubernetes Service (AKS). MaaS adds multi-tenant API governance on top of the inference platform, controlling authentication, rate limiting, and model access via Authorino and Limitador from Red Hat Connectivity Link (Kuadrant). A model is first deployed with a single LLMInferenceService YAML, creating a vLLM server and router-scheduler, but the endpoint is open. Applying a MaaSSubscription and MaaSAuthPolicy activates governance: Authorino validates API keys or JWTs, and Limitador enforces token rate limits. The post demonstrates three enforcement scenarios on an AKS cluster with a T4 GPU running Qwen3-0.6B: 401 for missing or invalid keys, 200 for authenticated requests, and 429 when rate limits are exceeded, blocking requests before they reach vLLM.
Sharding etcd on hosted control planes splits the key-value store into multiple instances by resource kind, a tech preview feature tested with a main shard for cluster state and a dedicated events shard on Red Hat OpenShift 4.21 with HyperShift. Under event pressure from synthetic storms and realistic crashlooping pods, the sharded configuration showed up to 96% fewer slow applies, faster compaction and raft snapshot durations, and 60% more WAL fsyncs and 90% more backend commits combined. The sharded arm never failed the event storm workload, while the standard arm occasionally did. Pod-ready latencies and workload finish times remained comparable across both configurations. The trade-offs are roughly 1.8 to 2.3 times the etcd container memory and a compaction regression under large custom resource quota pressure that is still under investigation.
Local developer workstations introduce three concrete operational risks: production-parity risk from environment drift, slow time-to-first-commit for new engineers, and unmanaged security exposure from misconfigured tools. Red Hat OpenShift Dev Spaces addresses these by moving compute and tooling into containerized workspaces running directly on the cluster, with environments declared as code in a devfile. A production deployment requires four-zone namespace isolation separating control (openshift-devspaces), core server (devspaces-checluster), secure asset (devspaces-openvsx), and sandbox (username-devspaces) zones to prevent operator reconciliation from overriding custom configurations. Resource governance uses a two-tier model with defaultContainerResources fallback limits and containerResourceCaps absolute per-container ceilings, while automated cleanup cron jobs (monthly on the 1st) and inactivity idling (after 4 hours) protect etcd health by removing stopped workspaces after 30 days. The guide concludes with next steps for authentication and private extension registries.
Network bandwidth is a compressible resource, so setting a flat cap wastes capacity when the link is idle. The real goal is to prioritise production traffic while guaranteeing background workloads a minimum floor, allowing either class to burst to full capacity when the other is idle. The upcoming NetworkQoS CRD in OVN-Kubernetes will bring declarative traffic management, but until then administrators can use Linux traffic control (tc). A hierarchical token bucket (HTB) qdisc tree on the node’s physical interface, with filters matching background traffic by destination IP, provides the desired dynamic prioritisation. In a lab test with 25 Gbps bonded NICs, production traffic maintained 19.5 Gbps under contention while background traffic was throttled to 2.7 Gbps, just above its guaranteed floor.
Red Hat’s migration toolkit for applications 8.3 is now generally available, adding agentic code modernization, direct storage mobility, and automated pipeline generation to OpenShift. The release serves as the official transition path from the migration toolkit for containers, which has reached end of support. For stateful workloads, it introduces direct TLS-encrypted persistent volume claim transfers with a stage-and-cutover flow, S3-mediated indirect transfers for disconnected environments, and intra-cluster StorageClass conversions that bypass Kubernetes immutability constraints. A developer preview agentic migration factory runs configurable multi-agent workflows that analyze dependencies, update code, and validate builds across independent stages, passing context through handoff files on shared Git branches. The toolkit also ships three new Helm-based generators that produce Tekton pipelines, Argo CD Application resources, and Kubernetes deployment manifests for onboarded applications.
GPU-accelerated Kubernetes clusters depend on compatible versions across host kernels, GPU drivers, container runtimes, networking, storage, operators, and workload frameworks, each on its own release cycle. A configuration that works for one service, GPU generation, and Kubernetes release may silently fail for another, and tracing version conflicts after deployment is slow. NVIDIA AI Cluster Runtime (AICR) v1.0 addresses this with version-locked, validated recipes that pin component combinations, render deployment artifacts for Helm, Argo CD, Flux, or Helmfile, and carry signed validation evidence from tested hardware. The release establishes a stable compatibility contract across its CLI, REST API, Go SDK, bundle layout, and artifact schemas, with merge-blocking checks on public interfaces. AICR now has over 100 contributors, with almost half from outside NVIDIA, and integrations from Pulumi Labs and Mirantis demonstrate consuming the same configuration through different tools.
Hosted control planes for Red Hat OpenShift, based on the HyperShift project, decouple the cluster control plane from worker nodes by running control-plane components as pods inside a central management cluster. On bare-metal infrastructure, OpenShift Virtualization provisions worker nodes as virtual machines, enabling live migration and reducing hardware requirements. The article presents a declarative GitOps workflow using Helm templates and an Argo CD ApplicationSet that dynamically generates deployments from tenant-specific YAML configuration files in Git. Prerequisites include a bare-metal management cluster with OpenShift Virtualization, Red Hat Advanced Cluster Management, and OpenShift GitOps, plus networking configured with NMState, Multus CNI, and MetalLB. The approach achieves a “Cluster-as-a-Service” model where provisioning a production-grade cluster requires only a Git push.
Agent sprawl arises when many AI agents on Red Hat OpenShift become redundant, insecure, and untraceable, using static API keys over HTTP and lacking visibility into reasoning failures. Rossoctl, a Kubernetes-native controller, automates discovery by declaring AgentRuntime resources that fetch metadata to create AgentCard custom resources, and uses a two-phase Helm deployment to resolve the URL dependency between routes and AgentCards. It adopts the A2A protocol, which standardizes agent interaction through a machine-readable AgentCard at /.well-known/agent-card.json and a JSON-RPC interface for consistent payloads. Security follows a six-layer zero-trust model: SPIRE issues short-lived certificates, Istio enforces mTLS, AgentCards are signed with JWS, and AuthBridge provides scoped OAuth2 tokens. Observability uses opt-in MLflow tracing that captures orchestration flow, tool calls, and model latency, enabling diagnosis of semantic failures such as hallucinations or data leaks that standard metrics miss.
A platform engineer traces why a non-root OpenShift container configured with CAP_NET_RAW still fails ping with Operation not permitted. The root cause is that SCC is only an admission-time gate that does not enforce runtime privileges. CRI-O translates the PodSpec into an OCI spec with the capability in the Bounding, Effective, and Permitted sets but deliberately omits ambient capabilities. During the runtime’s UID transition from root to the target non-root user, the kernel clears both Permitted and Effective sets, leaving an empty CapEff. The only mechanism that survives a UID transition for non-root processes is ambient capabilities, which CRI-O intentionally avoids to preserve the Kubernetes assumption that switching to a non-root user drops capabilities. To resolve this, file capabilities can be applied directly to the binary via setcap, avoiding the need for ambient capabilities.
Kube AuthKit is a lightweight Python library that unifies Kubernetes and OpenShift authentication behind a single API, eliminating conditional code paths across local, pod, notebook, and CI/CD environments. It supports four strategies—kubeconfig, in-cluster, OIDC, and OpenShift OAuth—auto-detected by an AuthFactory that probes environment variables and in-cluster markers in a defined order. The library handles token lifecycle transparently, with in-memory storage by default and optional OS keyring persistence for local development to avoid repeated logins. It provides two OIDC flows: device code for headless environments and authorization code with PKCE for interactive applications, both with automatic token refresh. Kube AuthKit returns a standard Kubernetes Python client ApiClient, ensuring full compatibility with the official library, and is actively maintained by the Open Data Hub community for OpenShift AI environments.
The Kubernetes device plugin model treats GPUs as anonymous integers, making fair sharing impossible because a full 80 GB GPU and a 5 GB partition both consume ‘1’ from the same quota. Dynamic resource allocation (DRA), generally available in Kubernetes 1.34, replaces this by having drivers advertise real device attributes like memory and compute capability, but DRA alone does not address quota management or preemption. Red Hat build of Kueue 1.4 on Red Hat OpenShift integrates with DRA to enforce quota, borrowing, and preemption for both whole GPUs and partitionable devices using device-count or counter-based memory accounting. For partitionable GPUs, Kueue reads counter data from DRA drivers and charges GPU memory against ClusterQueue quota instead of device count, enabling accurate fair sharing between training and inference workloads. Future developments include extended resources for simpler adoption without ResourceClaimTemplates, consumable capacity for dynamic time-slicing, and scheduler-library integration to verify scheduling feasibility.
Binary pass/fail uptime testing no longer suffices for cloud-native deployments because a cluster can remain operational while dropping packets, breaching SLOs, or suffering performance degradation. Krkn addresses this with a trifecta of telemetry: a weighted Resiliency Score, continuous HTTP Health Checks, and KubeVirt Checks. The Resiliency Score is a 0–100% macro-health metric that evaluates live Prometheus data against pre-defined SLOs using custom weights, allowing engineers to prioritize business-critical components like authentication or payment gateways. HTTP Health Checks probe application endpoints at configurable intervals, capturing exact downtime start and end times alongside HTTP status codes, enabling teams to track self-healing duration. KubeVirt Checks monitor virtual machine instances via real-time SSH connectivity and record node, IP address changes, and exact down-time, with a final post-chaos check that flags VMs remaining unreachable after the experiment concludes.
Managing edge infrastructure at scale across hundreds of locations demands consistent management rather than disparate tools. Red Hat’s layered approach organizes tools into three compartments: a universal foundation, OS-level fleet management for RHEL, and Kubernetes lifecycle management for OpenShift. The universal layer includes Ansible for agentless automation, Quay for container image registry with geo-replication, Lightspeed for proactive drift detection and compliance, and Trusted Software Supply Chain for end-to-end security. For RHEL-based edges, Edge Manager provides SaaS fleet management with OTA updates while Satellite offers on-premise content management for air-gapped environments. For OpenShift-based edges, Advanced Cluster Management handles cluster lifecycle and zero-touch provisioning while Advanced Cluster Security enforces container security across the fleet.
Red Hat conducted a performance and scale test of the argocd-agent pull architecture for GitOps, using 42 physical machines, 500 spoke clusters, and software including OpenShift 4.21 and OpenShift GitOps 1.20. The pull model decentralizes reconciliation to lightweight agents on each managed cluster, which pull configurations from a central hub, improving security by eliminating inbound administrative ports and reducing cross-cluster network traffic by up to 80%. In the bootstrap scenario, generating and syncing 31,200 applications across 480 clusters completed in 13 minutes 3 seconds, while the sync-to-commit scenario, which includes a polling delay, took 13 minutes. The test revealed bottlenecks including Kubernetes API limits on the hub, informer processing slowdowns causing watch restarts, event writer retry issues under load, and ApplicationSet controller RAM consumption. The team also deleted all 31,200 applications in 4 minutes 53 seconds using cascading deletion, and the insights are informing future OpenShift GitOps releases.
GPU clusters can pass standard health checks yet fail under real AI training workloads due to a single degraded GPU, network link, or misrouted traffic. NVIDIA Cluster Readiness Engine (NVCRE) is an open source Kubernetes controller that runs real distributed workloads across topology-aware node groups before production jobs land, measuring results and reporting which nodes failed each test. Its layered API uses Custom Resource Definitions for Certification, Workflow, and Job resources, attributing each failure to a specific node and category such as NCCL bandwidth or training goodput. The WorkloadRun API handles platform detection, NCCL settings, shared-memory volumes, and gang scheduling, so operators do not need to write those manifests by hand. NVCRE integrates with NVSentinel for passive telemetry and is available under Apache 2.0, with a roadmap for inference and automated lifecycle validation.
Managing Kubernetes node fleets, especially GPU nodes, is difficult because host-level changes can disrupt long-running training jobs, and traditional tools are not cluster-aware. NodeWright, an open-source Kubernetes-native package manager from NVIDIA, solves this by declaratively configuring and updating node operating systems while respecting PodDisruptionBudgets and non-interruptible workload labels. The operator orchestrates a sequence of cordon, wait, drain, apply, interrupt, and uncordon on each node, using container image packages that carry modifications such as kernel parameters, security agents, and verification scripts. It supports progressive rollouts across a fleet via DeploymentPolicy compartments with configurable disruption budgets and strategies like batch, canary, and rolling. NodeWright integrates with NVIDIA AI Cluster Runtime for validated configurations and is part of the DSX OS open-source portfolio, available under Apache 2.0.
The best engineering reads, once a week.
