Loading post…
MetaBy Arvind Srinivasan, Neil Spring, Omar Baldonado, Rajiv KrishnamurthyAnnouncement
MetaRoCE: A New RDMA Transport Built for AI-Scale Ethernet
Summary
Meta has designed MetaRoCE, a clean-sheet RDMA transport for AI workloads running on commodity Ethernet, targeting clusters that can reach million-GPU scale. Unlike standard RoCE, it treats packet loss and out-of-order delivery as normal: packets are sprayed across NIC-managed paths, written directly to their destinations, and retransmitted selectively without PFC or reorder buffers. It combines per-path telemetry and congestion windows with ECN-based sender control and receiver-driven fair-share hints, while a single connection supports multiple ordered streams and paths. On a 64-node AMD GPU cluster using RCCL collectives, Meta reports higher throughput and lower flow-completion times than RoCEv2, about 86% throughput at 1% packet loss, linear scaling across four- and eight-plane topologies, and autonomous recovery from plane failures. Meta is contributing the specification, reference implementation, and compliance suite to OCP, with a broader release planned for October at the 2026 OCP Global Summit.
Context
Frontier AI training and inference depend on communication between large numbers of GPUs. Collective operations are paced by the slowest transfer, while distributed inference is affected by communication latency. The post identifies standard RoCE’s reliance on in-order delivery, PFC, and limited packet spraying as constraints for large-scale, multiplane Ethernet networks.
Approach / What changed
MetaRoCE moves transport intelligence to the NIC endpoint. It uses packet spraying across NIC-managed paths, direct out-of-order placement, per-path telemetry and windows, selective retransmission over lossy Ethernet, ECN-based sender control, receiver-driven fair-share rate hints, and unified connections carrying multiple streams and paths. Meta is releasing the specification, a software reference implementation, and a compliance suite through OCP.
Takeaways
MetaRoCE operates without PFC or pause frames and treats Ethernet as lossy; a 256-bit selective acknowledgment bitvector identifies missing packets for immediate retransmission.
A single connection supports multiple independent ordered streams and many underlying paths, keeping congestion state from growing with workload parallelism.
In testing on a 64-node AMD GPU cluster, MetaRoCE maintained approximately 86% throughput at 1% packet loss and redistributed traffic autonomously during simulated plane failures.
Source and freshness
- Source
- engineering.fb.com
- Published
- Added to Yomu
- Last updated
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.