Topic

FSDP

1 post about FSDP. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Posts

Google
Integrating TorchTPU into TorchTitan and Optimizing Models for TPU
TorchTitan TPU is a fork of Meta’s TorchTitan that enables PyTorch model training on Google Cloud TPUs using the TorchTPU native backend with minimal device initialization changes. SimpleFSDP replaces FSDP2 hooks with DTensor parametrization, making all-gathers and reduce-scatters ordinary nodes in the traced graph for compiler scheduling. The Graph Trainer captures the full forward, loss, and backward step as one XLA program via make_fx, achieving 51-52% MFU on Qwen3-8B with functional AdamW. On Llama 3 405B, TorchTitan TPU reaches about 57% MFU on up to 1,024 chips, and a weak-scaling test of Qwen3 235B MoE retained 92% of per-chip throughput from 64 to 8,192 chips. The repository includes an optimization playbook, a coding-agent model-onboarding skill, and supports over 500 Hugging Face models through a Transformers backend.

The best engineering reads, once a week.

Topic

1 post about FSDP. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Google
Integrating TorchTPU into TorchTitan and Optimizing Models for TPU
TorchTitan TPU is a fork of Meta’s TorchTitan that enables PyTorch model training on Google Cloud TPUs using the TorchTPU native backend with minimal device initialization changes. SimpleFSDP replaces FSDP2 hooks with DTensor parametrization, making all-gathers and reduce-scatters ordinary nodes in the traced graph for compiler scheduling. The Graph Trainer captures the full forward, loss, and backward step as one XLA program via make_fx, achieving 51-52% MFU on Qwen3-8B with functional AdamW. On Llama 3 405B, TorchTitan TPU reaches about 57% MFU on up to 1,024 chips, and a weak-scaling test of Qwen3 235B MoE retained 92% of per-chip throughput from 64 to 8,192 chips. The repository includes an optimization playbook, a coding-agent model-onboarding skill, and supports over 500 Hugging Face models through a Transformers backend.

The best engineering reads, once a week.