Five Checkpoints to Bitwise-Deterministic Training Runs
A five-checkpoint workflow for making large-scale training runs bitwise reproducible: pick the guarantee, fingerprint the run, and trace divergence down to the kernel.
Loading post…
NVIDIAAshwath AithalExplainer
Scale Bitwise-Deterministic Pretraining with NVIDIA Megatron Core
Summary
Bitwise determinism ensures independent and checkpoint-resumed training runs follow the same numerical trajectory, providing four key benefits: replaying failures, preserving checkpoint continuity, detecting unexpected differences, and comparing changes without run-to-run variation. At trillion-parameter scale across thousands of GPUs, nondeterminism complicates failure reproduction and fix validation. NVIDIA Megatron Core targets two guarantees: independent reproducibility and bitwise checkpoint resume. The post describes a validation procedure using full-precision metrics and a progressive tracing workflow that uses torch.hash_tensor for GPU-resident fingerprints to narrow divergence from end-to-end metrics to the responsible kernel. A Nemotron case study demonstrates that kernel optimizations, such as separating grouped-GEMM writers and applying a fixed reduction order, can reduce the determinism performance overhead to approximately 2% steady-state determinism tax at scale.
Context
Large-scale pretraining of trillion-parameter models across thousands of GPUs introduces nondeterminism from multiple parallelism dimensions, low-precision computation, and distributed checkpointing. Bitwise determinism ensures independent and checkpoint-resumed runs follow the same numerical trajectory, enabling reliable failure replay, loss-spike debugging, system-change validation, and recovery from interrupted training. NVIDIA Megatron Core targets two guarantees: independent reproducibility and bitwise checkpoint resume. The post describes validation procedures, a progressive tracing workflow to localize divergence, and kernel optimizations to reduce performance overhead, illustrated by a Nemotron case study.
Approach / What changed
The approach involves validating determinism by fingerprinting full-precision metrics at every step and comparing across runs. When divergence occurs, a progressive tracing workflow narrows the scope from end-to-end metrics to the responsible kernel by comparing input and output fingerprints at each level. Kernel optimizations, such as separating grouped-GEMM writers and applying a fixed reduction order, reduce performance overhead while maintaining determinism. The Nemotron case study demonstrates these techniques at scale.
Takeaways
Bitwise determinism requires fixed data order, architecture, recipe, parallelism, software, runtime settings, and hardware to ensure independent and checkpoint-resumed runs follow the same numerical trajectory.
The validation procedure uses full-precision metrics and torch.hash_tensor fingerprints, not printed loss values, to detect mismatches; the first mismatching step helps narrow the investigation.
Kernel optimizations like separating grouped-GEMM writers and applying a fixed reduction order can reduce the determinism performance overhead to approximately 2% steady-state determinism tax at scale.
FAQ
How do you verify bitwise determinism?
The post recommends fingerprinting selected numerical metrics at every step, such as training loss and gradient norm, and comparing them across runs. Printed loss values are not sensitive enough; use torch.hash_tensor for GPU-resident fingerprints and byte-level comparisons to confirm bitwise equality.
What causes nondeterminism in large-scale pretraining?
Common sources include runtime autotuning selecting different kernel configurations, unfixed reduction order in fused auxiliary losses, incomplete RNG state restoration after checkpoint resume, low-level library kernel changes, and scale-dependent communication or expert-parallel paths.
How do you fix nondeterminism when it breaks?
Use a progressive tracing workflow that narrows divergence from end-to-end metrics to the responsible kernel by comparing input and output fingerprints at each scope. Classify each rank's first mismatch by causal role, rule out false alarms like tracing blind spots, and validate the fix with paired-run checks.
Source and freshness
- Source
- developer.nvidia.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
Visual explainers
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.