---
title: "Boost Training Goodput: How Continuous Checkpointing Optimizes Reliability in Orbax and MaxText"
description: "Orbax and MaxText introduce continuous checkpointing to balance training reliability and performance when hardware failures or preemption events interrupt long-running jobs. Fixed checkpoint intervals create a trade-off: infrequent saves risk substantial lost work, while frequent saves can block or bottleneck training, particularly over unstable networks. MaxText enables asynchronous checkpointing that attempts a new save only after the previous background request is fulfilled; its continuous mode overrides checkpoint_period, and max_num_checkpoints_to_keep can limit storage use. A llama-3.1-70B benchmark on two v5p-128 slices found smaller P50 checkpoint intervals, but average training step time increased because of more frequent device-to-host transfers; the analysis estimated resource conservation against mean-time-between-failure assumptions. Orbax also supports minimum intervals and customizable preservation policies, while co-locating storage with training is recommended because cross-metro bandwidth can reduce checkpointing speed and reliability."
---

# Boost Training Goodput: How Continuous Checkpointing Optimizes Reliability in Orbax and MaxText

[Google](https://yomu.fyi/company/google) · Shutong Li, Colin Gaffney · Mar 31, 2026

**Type:** Benchmark

## Summary

Orbax and MaxText introduce continuous checkpointing to balance training reliability and performance when hardware failures or preemption events interrupt long-running jobs. Fixed checkpoint intervals create a trade-off: infrequent saves risk substantial lost work, while frequent saves can block or bottleneck training, particularly over unstable networks. MaxText enables asynchronous checkpointing that attempts a new save only after the previous background request is fulfilled; its continuous mode overrides checkpoint\_period, and max\_num\_checkpoints\_to\_keep can limit storage use. A llama-3.1-70B benchmark on two v5p-128 slices found smaller P50 checkpoint intervals, but average training step time increased because of more frequent device-to-host transfers; the analysis estimated resource conservation against mean-time-between-failure assumptions. Orbax also supports minimum intervals and customizable preservation policies, while co-locating storage with training is recommended because cross-metro bandwidth can reduce checkpointing speed and reliability.

## Context

Fixed checkpoint intervals force a difficult balance between preserving progress after hardware failures or preemption and avoiding training slowdowns from frequent, resource-intensive saves. The source also identifies network bandwidth, especially across metros and in multi-slice training, as a factor affecting checkpointing effectiveness.

## Approach / What changed

Continuous checkpointing uses asynchronous saves and starts another checkpoint only after the preceding save request is fulfilled. MaxText exposes configuration flags for the feature and storage retention, while Orbax provides minimum-interval saving and customizable preservation policies. The design keeps substantial checkpoint communication between storage and a single slice asynchronous.

## Takeaways

- Continuous checkpointing produced smaller P50 checkpoint intervals than checkpointing every 100 steps in the described llama-3.1-70B benchmark, but increased average training step time because of more frequent device-to-host transfers.
- In MaxText, continuous checkpointing overrides checkpoint\_period and attempts a save after the previous background saving request is fulfilled; max\_num\_checkpoints\_to\_keep limits retained storage.
- Orbax supports minimum intervals and custom preservation logic, while co-locating storage with the training cluster helps avoid cross-metro bandwidth degrading checkpoint speed and reliability.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance), [Reliability](https://yomu.fyi/topic/reliability)

- Source: [Google](https://developers.googleblog.com/boost-training-goodput-how-continuous-checkpointing-optimizes-reliability-in-orbax-and-maxtext)
- Source URL: https://developers.googleblog.com/boost-training-goodput-how-continuous-checkpointing-optimizes-reliability-in-orbax-and-maxtext
- Ingested by Yomu: 2026-09-09T14:28:30.557Z

[Read original post](https://developers.googleblog.com/boost-training-goodput-how-continuous-checkpointing-optimizes-reliability-in-orbax-and-maxtext)