Large-model training with SPMD struggles when jobs become non-uniform, such as pipeline parallelism or RL training loops, and cannot survive hardware failures without restarting. Pathways addresses this by introducing a single Python client that describes the entire computation as a graph of blocks and tensor edges. A distributed runtime places blocks on virtual devices, moves and reshards tensors, orders operations across the cluster, and remaps around a failed host. This separates model building from execution, allowing reasoning about the whole cluster instead of one slice. Pathways runs JAX programs on TPUs and is available on Google Cloud.
Context
Large-model training typically uses SPMD (Single Program, Multiple Data), where every accelerator runs the same program in lockstep and coordinates through collectives like AllReduce. This model struggles with non-uniform jobs such as pipeline parallelism or RL training loops, and it cannot survive hardware failures without restarting from a checkpoint.
Approach / What changed
Pathways introduces a single Python client that describes the entire computation as a graph of blocks and tensor edges. A distributed runtime places blocks on virtual devices, moves and reshards tensors, orders operations across the cluster, and remaps around failed hosts, separating model building from execution.
Takeaways
1
SPMD requires every accelerator to run the same program in lockstep, which becomes suboptimal for pipeline parallelism or RL training loops where stages have different dependencies.
2
Pathways uses a single client to describe the whole computation as a graph, and a distributed runtime handles placement, tensor movement, and ordering across the cluster.
3
Pathways can recover from a failed host by remapping around it, unlike SPMD which must restart the entire job from the last checkpoint.
FAQ
How does Pathways differ from SPMD?
Pathways separates model building from execution. You describe the whole computation as a graph from a single client, and a distributed runtime handles placement, tensor movement, and ordering. SPMD requires each process to manage its own devices and dependencies through collectives, and it cannot recover from a single failed machine without restarting.
How does Pathways relate to Ray?
Ray orchestrates Python AI workloads across clusters, scheduling tasks and actors, but the training inside each worker still runs as SPMD. Pathways changes the execution model underneath so one client coordinates every chip. They work at different layers and are complementary.
How does Pathways avoid deadlocks?
Pathways uses a central scheduler that puts every program's operations into one consistent order, so operations that must communicate start together instead of deadlocking. This prevents stalls when a group has more than one operation ready, such as forward and backward passes in pipelined training.
An AI runtime is the execution layer between model code and hardware: it feeds data to accelerators, places and reshards tensors, schedules work, and holds latency steady. Here are its four core jobs.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.