---
title: "Five Checkpoints to Bitwise-Deterministic Training Runs"
description: "A five-checkpoint workflow for making large-scale training runs bitwise reproducible: pick the guarantee, fingerprint the run, and trace divergence down to the kernel."
---

# Five Checkpoints to Bitwise-Deterministic Training Runs

**Type:** [How to](https://yomu.fyi/visual-explainers/category/how-to.md)

![A five-step vertical flow diagram of numbered checkpoint cards, from choosing a determinism guarantee through tensor fingerprinting, full-precision comparison, progressive divergence tracing, and measuring the determinism tax, with a side rail showing the evidence narrowing from metrics to kernel.](https://yomufyi.s3.us-east-1.amazonaws.com/visual-explainers/7a004561-3f48-473e-9b6e-ab734e6b863d/image-82515ab1-0804-4b26-ad03-62c9d2de9746.webp)

> A five-checkpoint workflow for making large-scale training runs bitwise reproducible: pick the guarantee, fingerprint the run, and trace divergence down to the kernel.

## Explanation

Bitwise determinism means two runs of the same job follow the same numerical trajectory, so a resumed run produces identical tensors, not merely similar metrics. This explainer turns it into five checkpoints: choose your guarantee, fingerprint GPU tensors with torch.hash\_tensor, compare at full precision, then trace divergence from end-to-end metrics down to the responsible kernel. The last checkpoint is a fix plus a measurement: enforce a fixed reduction order, re-verify, and check the overhead, which can land near a 2% steady-state determinism tax at scale. Nondeterminism across thousands of GPUs makes failures hard to replay and fixes hard to validate.

## FAQ

### What is the difference between independent reproducibility and bitwise checkpoint resume?

Independent reproducibility means two runs launched from the same starting point follow the same numerical trajectory. Bitwise checkpoint resume means a run restarted from a checkpoint continues along the trajectory of the uninterrupted run. NVIDIA Megatron Core targets both guarantees, because either one can break at trillion-parameter scale across thousands of GPUs.

### Why fingerprint tensors instead of just comparing loss or evaluation metrics?

End-to-end metrics can look close or even identical while the underlying trajectory has already diverged. Using torch.hash\_tensor gives a GPU-resident fingerprint of a tensor, so you can compare states directly and bisect the difference. The workflow starts with full-precision metrics, then progressively narrows from end-to-end numbers to the layer and finally the responsible kernel.

**Tags:** [Bitwise Determinism](https://yomu.fyi/topic/bitwise-determinism.md), [Debugging](https://yomu.fyi/topic/debugging.md), [Determinism](https://yomu.fyi/topic/determinism.md), [GPU](https://yomu.fyi/topic/gpu.md), [Multi-GPU](https://yomu.fyi/topic/multi-gpu.md), [Testing](https://yomu.fyi/topic/testing.md)

## Related reading

### [How California built its Behavioral Health Public County Profile on Databricks](https://yomu.fyi/post/how-california-built-its-behavioral-health-public-county-profile-on-da.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Michael Smith, Cameron Kahn, Stephen Hsu, Scott Black, Ben McKay, Varun Ganti
- Published: Oct 10, 2026
- Type: Explainer

The California Department of Health Care Services built the Behavioral Health Public County Profile on Databricks, a public dashboard consolidating behavioral health data from all 58 counties for nearly 40 million Californians. Launched on October 1, 2026, following the 2024 passage of Proposition 1, the dashboard supports the new Behavioral Health Services Act (BHSA) tying funding to statewide goals for outcomes, equity, and transparency. The platform uses Spark Declarative Pipelines and Lakeflow Connect for unified data ingestion, Semantic Views for standardized metrics across counties, and serverless compute to process data at scale without infrastructure management. It operates on a FedRAMP-authorized environment meeting federal security standards for health data, with data quality and lineage ensuring every number can be traced to its source. The County Profile enables public comparison of county plans, funding, service delivery, and performance measures, creating a feedback loop between policy intent and measurable outcomes for democratic oversight.


### [UI freeze reports are now available in JetBrains Marketplace](https://yomu.fyi/post/ui-freeze-reports-are-now-available-in-jetbrains-marketplace.md)
- Company: [Jetbrains](https://yomu.fyi/company/jetbrains.md)
- Author: Jakub Chrzanowski
- Published: Oct 9, 2026
- Type: Announcement

JetBrains Marketplace now provides UI freeze reports for plugin authors, accessible from the Freezes tab after enabling Marketplace reporting via the JetBrainsMarketplaceErrorReportSubmitter handler in plugin.xml. Reports are grouped by error type, with each pile showing a representative report and a thread-dump excerpt that identifies threads involved, such as the event dispatch thread (EDT), helping spot recurring patterns. Full thread dumps captured every five seconds during the freeze are available, along with optional CPU snapshots and performance metrics, and comparing dumps across samples can reveal persistent operations. The analyze-freeze skill guides AI agents through investigating IntelliJ Platform thread dumps, focusing on the EDT, coroutine behavior, and common patterns like non-cancellable read actions blocking the UI, with suggestions for fixes. Before fixing, authors should check the plugin version, keep blocking I/O outside read actions, and test changes by repeating the operation and examining fresh diagnostic evidence.


### [Build a multi-tenant platform with OpenShift Dev Spaces](https://yomu.fyi/post/build-a-multi-tenant-platform-with-openshift-dev-spaces.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Ajay Kanse
- Published: Oct 9, 2026
- Type: Tutorial

Local developer workstations introduce three concrete operational risks: production-parity risk from environment drift, slow time-to-first-commit for new engineers, and unmanaged security exposure from misconfigured tools. Red Hat OpenShift Dev Spaces addresses these by moving compute and tooling into containerized workspaces running directly on the cluster, with environments declared as code in a devfile. A production deployment requires four-zone namespace isolation separating control (openshift-devspaces), core server (devspaces-checluster), secure asset (devspaces-openvsx), and sandbox (username-devspaces) zones to prevent operator reconciliation from overriding custom configurations. Resource governance uses a two-tier model with defaultContainerResources fallback limits and containerResourceCaps absolute per-container ceilings, while automated cleanup cron jobs (monthly on the 1st) and inactivity idling (after 4 hours) protect etcd health by removing stopped workspaces after 30 days. The guide concludes with next steps for authentication and private extension registries.


### [Introducing on-demand CPU and memory profiling with flamegraphs for Workers and Durable Objects](https://yomu.fyi/post/introducing-on-demand-cpu-and-memory-profiling-with-flamegraphs-for-wo.md)
- Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md)
- Author: Dominik Picheta
- Published: Oct 9, 2026
- Type: Announcement

Cloudflare introduced on-demand CPU and memory profiling for Workers and Durable Objects, accessible via the CLI or the Dashboard’s Observability page. Users can generate interactive flamegraphs and download pprof files, with each rectangle’s width representing CPU time or memory usage over a chosen duration. Profiling the R2 binding Worker revealed that genericR2JsonReplacer, called by JSON.stringify, processed nested values multiple times; fixing it made the function 2.7x faster. A different Worker with P999 memory at 133 MB (above the 128 MB limit) was found to have a partially enabled Prometheus code path accounting for 66.7% of allocations; disabling it reduced P999 memory to 118 MB and eliminated out-of-memory errors. The profiling system locates the correct data center and isolate for the Worker or Durable Object, and continuous profiling is being developed to automatically capture samples without manual initiation.


### [Scale Bitwise-Deterministic Pretraining with NVIDIA Megatron Core](https://yomu.fyi/post/scale-bitwise-deterministic-pretraining-with-nvidia-megatron-core.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Ashwath Aithal
- Published: Oct 6, 2026
- Type: Explainer

Bitwise determinism ensures independent and checkpoint-resumed training runs follow the same numerical trajectory, providing four key benefits: replaying failures, preserving checkpoint continuity, detecting unexpected differences, and comparing changes without run-to-run variation. At trillion-parameter scale across thousands of GPUs, nondeterminism complicates failure reproduction and fix validation. NVIDIA Megatron Core targets two guarantees: independent reproducibility and bitwise checkpoint resume. The post describes a validation procedure using full-precision metrics and a progressive tracing workflow that uses torch.hash\_tensor for GPU-resident fingerprints to narrow divergence from end-to-end metrics to the responsible kernel. A Nemotron case study demonstrates that kernel optimizations, such as separating grouped-GEMM writers and applying a fixed reduction order, can reduce the determinism performance overhead to approximately 2% steady-state determinism tax at scale.