---
title: "Manage Kubernetes Node Fleets with NodeWright"
description: "Managing Kubernetes node fleets, especially GPU nodes, is difficult because host-level changes can disrupt long-running training jobs, and traditional tools are not cluster-aware. NodeWright, an open-source Kubernetes-native package manager from NVIDIA, solves this by declaratively configuring and updating node operating systems while respecting PodDisruptionBudgets and non-interruptible workload labels. The operator orchestrates a sequence of cordon, wait, drain, apply, interrupt, and uncordon on each node, using container image packages that carry modifications such as kernel parameters, security agents, and verification scripts. It supports progressive rollouts across a fleet via DeploymentPolicy compartments with configurable disruption budgets and strategies like batch, canary, and rolling. NodeWright integrates with NVIDIA AI Cluster Runtime for validated configurations and is part of the DSX OS open-source portfolio, available under Apache 2.0."
---

# Manage Kubernetes Node Fleets with NodeWright

[NVIDIA](https://yomu.fyi/company/nvidia-developer-blog) · Michelle Horton · Sep 23, 2026

**Type:** Problem & solution

## Summary

Managing Kubernetes node fleets, especially GPU nodes, is difficult because host-level changes can disrupt long-running training jobs, and traditional tools are not cluster-aware. NodeWright, an open-source Kubernetes-native package manager from NVIDIA, solves this by declaratively configuring and updating node operating systems while respecting PodDisruptionBudgets and non-interruptible workload labels. The operator orchestrates a sequence of cordon, wait, drain, apply, interrupt, and uncordon on each node, using container image packages that carry modifications such as kernel parameters, security agents, and verification scripts. It supports progressive rollouts across a fleet via DeploymentPolicy compartments with configurable disruption budgets and strategies like batch, canary, and rolling. NodeWright integrates with NVIDIA AI Cluster Runtime for validated configurations and is part of the DSX OS open-source portfolio, available under Apache 2.0.

## Context

Kubernetes node management, especially for GPU clusters, is challenging because host-level changes can disrupt long-running training jobs. Traditional tools like Ansible are not cluster-aware and do not respect Kubernetes primitives. NodeWright is an open-source Kubernetes-native package manager that addresses this by declaratively configuring nodes while respecting PodDisruptionBudgets and non-interruptible workloads.

## Approach / What changed

NodeWright uses a Kubernetes operator and custom resources to manage host-level changes. It orchestrates a sequence of cordon, wait, drain, apply, interrupt, and uncordon on each node. Packages are container images with modifications and verification scripts. Progressive rollouts are controlled via DeploymentPolicy compartments with disruption budgets and strategies. It integrates with NVIDIA AI Cluster Runtime for validated configurations.

## Takeaways

- NodeWright is a Kubernetes-native package manager that uses an operator to cordon, wait, drain, apply, interrupt, and uncordon nodes, respecting PodDisruptionBudgets and non-interruptible workload labels.
- Packages are container images that carry modifications and verification scripts; they can set sysctl, GRUB parameters, install agents, remediate CVEs, and more, with built-in validation.
- NodeWright supports progressive rollouts across a fleet using DeploymentPolicy compartments, each with its own disruption budget and rollout strategy (e.g., batch, canary, rolling).

## FAQ

### How does NodeWright avoid disrupting running training jobs?

NodeWright cordons the node to prevent new workloads, waits for non-interruptible workloads to finish, drains evictable pods while honoring PodDisruptionBudgets, applies the package, interrupts if needed (e.g., reboot), and then uncordons the node. This sequence ensures training jobs are not disrupted.

### What host-level changes can NodeWright packages perform?

NodeWright packages can set sysctl and GRUB parameters, configure crash dump collection, create logical volumes, install security agents, remediate CVEs, and perform other system-level tasks. Each package includes verification scripts to validate the change.

### How does NodeWright integrate with existing Kubernetes tooling?

NodeWright custom resources are deployed via kubectl, Helm, Argo CD, Flux, or any GitOps tooling. It respects existing Kubernetes primitives like PodDisruptionBudgets, node selectors, taints, and tolerations, integrating seamlessly into existing workflows.

**Tags:** [GPU](https://yomu.fyi/topic/gpu), [Kubernetes](https://yomu.fyi/topic/kubernetes), [NVIDIA](https://yomu.fyi/topic/nvidia), [Open Source](https://yomu.fyi/topic/open-source)

- Source: [NVIDIA](https://developer.nvidia.com/blog/manage-kubernetes-node-fleets-with-nodewright)
- Source URL: https://developer.nvidia.com/blog/manage-kubernetes-node-fleets-with-nodewright
- Ingested by Yomu: 2026-09-23T21:00:11.083Z

[Read original post](https://developer.nvidia.com/blog/manage-kubernetes-node-fleets-with-nodewright)