---
title: "New in Together GPU Clusters: Reliability and control for production GPU clusters"
description: "Together GPU Clusters introduces changes for production training and inference workloads, addressing hardware failures, scheduler problems, and the limits of single-admin access. Platform health improvements include continuously running passive health checks, human-approved node repair recommendations, and Slurm-on-Kubernetes 2.0, which adds self-healing daemons, durable job accounting, descendant-process cleanup, and refreshed GPU state after reschedules. Operational controls include a cluster details view with health, utilization, events, configuration, and node actions, plus external OIDC, startup scripts, and an acceptance-test opt-out. Newly provisioned Slurm clusters use the latest stack by default, while existing managed clusters can be migrated during a maintenance window at no cost; the stated result is faster recovery, fewer support tickets, and greater visibility and control for growing teams running large clusters."
---

# New in Together GPU Clusters: Reliability and control for production GPU clusters

[Together AI](https://yomu.fyi/company/together-ai) · Jul 15, 2026

**Type:** Announcement

## Summary

Together GPU Clusters introduces changes for production training and inference workloads, addressing hardware failures, scheduler problems, and the limits of single-admin access. Platform health improvements include continuously running passive health checks, human-approved node repair recommendations, and Slurm-on-Kubernetes 2.0, which adds self-healing daemons, durable job accounting, descendant-process cleanup, and refreshed GPU state after reschedules. Operational controls include a cluster details view with health, utilization, events, configuration, and node actions, plus external OIDC, startup scripts, and an acceptance-test opt-out. Newly provisioned Slurm clusters use the latest stack by default, while existing managed clusters can be migrated during a maintenance window at no cost; the stated result is faster recovery, fewer support tickets, and greater visibility and control for growing teams running large clusters.

## Context

Published July 15, 2026, the post presents changes to Together GPU Clusters for teams running multi-day training and production inference workloads at scale. The work is organized around platform health and operational control.

## Approach / What changed

The platform combines passive health checks with human-approved node repair recommendations covering reboot, reprovision, failover, and removal. Its Slurm-on-Kubernetes stack, based on a fork of Slinky, supervises worker daemons, reaps orphaned processes, stores accounting on durable PVC-backed storage, cleans up job descendants, and rebuilds GPU state after node starts. Operational additions include a cluster details view, external OIDC, startup scripts, and an acceptance-test opt-out.

## Takeaways

- Passive health checks continuously observe workloads, logs, and metrics across cluster nodes, covering GPU bus failures, thermal throttling, Xid errors, and Slurm node drains with near-zero overhead on running jobs.
- Auto node repair maps detected failure signatures to reboot, reprovision, failover, or removal, but an operator approves the recommendation before Together performs graceful recovery.
- The updated Slurm stack preserves sacct history on PVC-backed storage, cleans up descendant processes at job end, and rebuilds GPU state after rescheduling so scheduling reflects the actual hardware.

## FAQ

### How do passive health checks differ from active health checks?

Active checks exercise hardware against a known-good baseline and are useful during provisioning or on idle nodes. Passive checks run continuously while workloads are active, using logs and metrics to detect degradation with near-zero overhead.

### What happens after a node-level issue is detected?

The monitoring system recommends a remediation based on the failure signature. An operator reviews and approves the action, after which Together handles the recovery through reboot, reprovision, failover, or removal.

### How does the new Slurm stack avoid stale or lost cluster state?

Job accounting is stored on durable PVC-backed storage, so pod restarts and reschedules do not erase it. Slurm also rebuilds its GPU view when a node starts, while kernel-level descendant tracking cleans up processes and their resources after jobs end.

**Tags:** [GPU](https://yomu.fyi/topic/gpu), [Kubernetes](https://yomu.fyi/topic/kubernetes), [Monitoring](https://yomu.fyi/topic/monitoring), [OIDC](https://yomu.fyi/topic/oidc)

- Source: [Together AI](https://www.together.ai/blog/new-in-together-gpu-clusters-reliability-and-control-for-production-gpu-clusters)
- Source URL: https://www.together.ai/blog/new-in-together-gpu-clusters-reliability-and-control-for-production-gpu-clusters
- Ingested by Yomu: 2026-09-19T10:13:40.302Z

[Read original post](https://www.together.ai/blog/new-in-together-gpu-clusters-reliability-and-control-for-production-gpu-clusters)