Loading…
How we keep GPUs reliable across Databricks AI
Steven Chen, Feng Wang, Bhavik Soni, Chengguang Yang, Albert Zhong, Naren Loganathan, Harsh Panchal, Jianwei Xie
- Source
- Databricks
- Published
- Added to Yomu
Summary
Databricks AI describes GPU reliability as a fleet-scale problem involving crashed jobs, silent slowdowns, and numerical corruption during distributed training. Its approach combines stress testing with demanding reinforcement learning, agentic coding, and document intelligence workloads, plus gpu-monitor, a multi-stage service that checks nodes during bootstrap, continuously while active, and periodically between workloads. The checks cover compute burn-in, peer connectivity, NCCL correctness and bandwidth, RDMA, ECC and HBM memory, PCIe integrity, throttling, thermal conditions, XID errors, and fabric health. A production incident showed why cumulative InfiniBand port downtime matters: a single outage exceeded the lower-level NCCL_IB_TIMEOUT, killing a collective before the longer PyTorch watchdog timeout. Nodes that fail checks are quarantined, while inter-node probes sweep message sizes from 8 bytes to 2 GiB and apply latency or BusBW criteria; the system is continuously expanded as new failure modes emerge.
Context
Distributed training jobs use large GPU fleets where hardware, fabric, and software failures are expected during runs. The difficult cases include silent performance degradation and numerical corruption, while crash logs such as NCCL watchdog timeouts often expose symptoms without identifying the underlying cause.
Approach / What changed
Databricks AI uses diverse, demanding training workloads to stress the platform and operates gpu-monitor across the node lifecycle. Bootstrap checks validate nodes before workloads, passive checks watch active nodes for emerging faults, and periodic multi-node tests validate inter-node fabric behavior. Failed nodes are cordoned, drained, quarantined, reset, and retested.
Takeaways
- GPU failures are grouped into crashed jobs, silent slowdowns, and numerical corruption; slowdowns can waste compute while logs and loss curves appear normal.
- gpu-monitor combines active bootstrap checks, passive continuous checks, and periodic multi-node fabric tests, including NCCL probes across payloads from 8 bytes to 2 GiB.
- A single sufficiently long InfiniBand port flap can outlast NCCL_IB_TIMEOUT and kill a collective, so health decisions use cumulative downtime rather than flap count.