Stop wasting GPU allocation in Kubernetes with GPU-pruner

Summary

GPU-pruner addresses idle GPU allocation in Kubernetes by identifying workloads whose hardware activity remains below a configured threshold, rather than relying on browser sessions, network traffic, or pod lifetime. It queries Prometheus for NVIDIA Data Center GPU Manager metrics, including engine activity and GPU utilization, and treats peak performance below 1% throughout a default 35-minute observation window as idle. After tracing an idle pod through ownerReferences or KServe labels, it scales the parent resource to zero, preserving workload metadata for later restoration. Slack notifications, grace periods, lookback checks, and dry-run mode provide safeguards, while Prometheus label handling must preserve the workload namespace and pod identity in DCGM telemetry.

Context

Kubernetes GPU workloads can remain allocated for days while showing almost no GPU engine activity, accumulating costs and blocking other users. Existing notebook and session cullers generally observe browser activity, requests, traffic, or pod lifetime rather than hardware utilization, so they can miss workloads that reserve GPUs without actively processing.

Approach / What changed

GPU-pruner queries NVIDIA DCGM telemetry through Prometheus and applies a configurable inactivity threshold, defaulting to 1% peak engine performance over 35 minutes. It traces idle pods to parent Kubernetes or ML resources, optionally notifies owners through Slack, and scales those parents to zero instead of deleting them. Dry-run mode, lookback validation, grace periods, and Prometheus label configuration support safer deployment.

Takeaways

• GPU-pruner monitors DCGM_FI_PROF_GR_ENGINE_ACTIVE and DCGM_FI_DEV_GPU_UTIL through PromQL, using hardware telemetry to define inactivity.
• The controller follows ownerReferences or KServe-specific labels from an idle pod to resources such as Deployments, StatefulSets, Kubeflow Notebooks, and InferenceServices.
• Scaling the parent resource to zero preserves its configuration and metadata, allowing the workload to be restored by scaling it back up or using the platform UI.

Related reading