---
title: "What is Pathways? Single-controller training for large models"
description: "Large-model training with SPMD struggles when jobs become non-uniform, such as pipeline parallelism or RL training loops, and cannot survive hardware failures without restarting. Pathways addresses this by introducing a single Python client that describes the entire computation as a graph of blocks and tensor edges. A distributed runtime places blocks on virtual devices, moves and reshards tensors, orders operations across the cluster, and remaps around a failed host. This separates model building from execution, allowing reasoning about the whole cluster instead of one slice. Pathways runs JAX programs on TPUs and is available on Google Cloud."
---

# What is Pathways? Single-controller training for large models

[Google](https://yomu.fyi/company/google) · Sujeeth Jinesh, Shaurya Gupta, Luke Baumann, Monica Song, Ivan Nardini, Akanksha Gupta, Anthony Owusu, Jing Yang, Sadi Kneipp, Yu Li · Nov 9, 2026

**Type:** Explainer

## Summary

Large-model training with SPMD struggles when jobs become non-uniform, such as pipeline parallelism or RL training loops, and cannot survive hardware failures without restarting. Pathways addresses this by introducing a single Python client that describes the entire computation as a graph of blocks and tensor edges. A distributed runtime places blocks on virtual devices, moves and reshards tensors, orders operations across the cluster, and remaps around a failed host. This separates model building from execution, allowing reasoning about the whole cluster instead of one slice. Pathways runs JAX programs on TPUs and is available on Google Cloud.

## Context

Large-model training typically uses SPMD (Single Program, Multiple Data), where every accelerator runs the same program in lockstep and coordinates through collectives like AllReduce. This model struggles with non-uniform jobs such as pipeline parallelism or RL training loops, and it cannot survive hardware failures without restarting from a checkpoint.

## Approach / What changed

Pathways introduces a single Python client that describes the entire computation as a graph of blocks and tensor edges. A distributed runtime places blocks on virtual devices, moves and reshards tensors, orders operations across the cluster, and remaps around failed hosts, separating model building from execution.

## Takeaways

- SPMD requires every accelerator to run the same program in lockstep, which becomes suboptimal for pipeline parallelism or RL training loops where stages have different dependencies.
- Pathways uses a single client to describe the whole computation as a graph, and a distributed runtime handles placement, tensor movement, and ordering across the cluster.
- Pathways can recover from a failed host by remapping around it, unlike SPMD which must restart the entire job from the last checkpoint.

## FAQ

### How does Pathways differ from SPMD?

Pathways separates model building from execution. You describe the whole computation as a graph from a single client, and a distributed runtime handles placement, tensor movement, and ordering. SPMD requires each process to manage its own devices and dependencies through collectives, and it cannot recover from a single failed machine without restarting.

### How does Pathways relate to Ray?

Ray orchestrates Python AI workloads across clusters, scheduling tasks and actors, but the training inside each worker still runs as SPMD. Pathways changes the execution model underneath so one client coordinates every chip. They work at different layers and are complementary.

### How does Pathways avoid deadlocks?

Pathways uses a central scheduler that puts every program's operations into one consistent order, so operations that must communicate start together instead of deadlocking. This prevents stalls when a group has more than one operation ready, such as forward and backward passes in pipelined training.

**Tags:** [Distributed Systems](https://yomu.fyi/topic/distributed-systems), [Google Cloud](https://yomu.fyi/topic/gcp), [JAX](https://yomu.fyi/topic/jax), [LLMs](https://yomu.fyi/topic/llm), [Pathways](https://yomu.fyi/topic/pathways)

- Source: [Google](https://developers.googleblog.com/what-is-pathways-single-controller-training-for-large-models)
- Source URL: https://developers.googleblog.com/what-is-pathways-single-controller-training-for-large-models
- Ingested by Yomu: 2026-11-09T12:00:20.898Z

[Read original post](https://developers.googleblog.com/what-is-pathways-single-controller-training-for-large-models)
