New in Together GPU Clusters: Reliability and control for production GPU clusters

Summary

Together GPU Clusters introduces changes for production training and inference workloads, addressing hardware failures, scheduler problems, and the limits of single-admin access. Platform health improvements include continuously running passive health checks, human-approved node repair recommendations, and Slurm-on-Kubernetes 2.0, which adds self-healing daemons, durable job accounting, descendant-process cleanup, and refreshed GPU state after reschedules. Operational controls include a cluster details view with health, utilization, events, configuration, and node actions, plus external OIDC, startup scripts, and an acceptance-test opt-out. Newly provisioned Slurm clusters use the latest stack by default, while existing managed clusters can be migrated during a maintenance window at no cost; the stated result is faster recovery, fewer support tickets, and greater visibility and control for growing teams running large clusters.

Takeaways

1. 1

   Passive health checks continuously observe workloads, logs, and metrics across cluster nodes, covering GPU bus failures, thermal throttling, Xid errors, and Slurm node drains with near-zero overhead on running jobs.

2. 2

   Auto node repair maps detected failure signatures to reboot, reprovision, failover, or removal, but an operator approves the recommendation before Together performs graceful recovery.

3. 3

   The updated Slurm stack preserves sacct history on PVC-backed storage, cleans up descendant processes at job end, and rebuilds GPU state after rescheduling so scheduling reflects the actual hardware.

FAQ