Topic

Megatron Core

1 post about Megatron Core. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Posts

NVIDIA
Scale Bitwise-Deterministic Pretraining with NVIDIA Megatron Core
Bitwise determinism ensures independent and checkpoint-resumed training runs follow the same numerical trajectory, providing four key benefits: replaying failures, preserving checkpoint continuity, detecting unexpected differences, and comparing changes without run-to-run variation. At trillion-parameter scale across thousands of GPUs, nondeterminism complicates failure reproduction and fix validation. NVIDIA Megatron Core targets two guarantees: independent reproducibility and bitwise checkpoint resume. The post describes a validation procedure using full-precision metrics and a progressive tracing workflow that uses torch.hash_tensor for GPU-resident fingerprints to narrow divergence from end-to-end metrics to the responsible kernel. A Nemotron case study demonstrates that kernel optimizations, such as separating grouped-GEMM writers and applying a fixed reduction order, can reduce the determinism performance overhead to approximately 2% steady-state determinism tax at scale.

The best engineering reads, once a week.

Topic

1 post about Megatron Core. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

NVIDIA
Scale Bitwise-Deterministic Pretraining with NVIDIA Megatron Core
Bitwise determinism ensures independent and checkpoint-resumed training runs follow the same numerical trajectory, providing four key benefits: replaying failures, preserving checkpoint continuity, detecting unexpected differences, and comparing changes without run-to-run variation. At trillion-parameter scale across thousands of GPUs, nondeterminism complicates failure reproduction and fix validation. NVIDIA Megatron Core targets two guarantees: independent reproducibility and bitwise checkpoint resume. The post describes a validation procedure using full-precision metrics and a progressive tracing workflow that uses torch.hash_tensor for GPU-resident fingerprints to narrow divergence from end-to-end metrics to the responsible kernel. A Nemotron case study demonstrates that kernel optimizations, such as separating grouped-GEMM writers and applying a fixed reduction order, can reduce the determinism performance overhead to approximately 2% steady-state determinism tax at scale.

The best engineering reads, once a week.