MTIA 300: Meta’s First Training Chip with Built-in NICs and Communication-Offloading Engines

Summary

MTIA 300 is Meta’s first in-house training and inference accelerator optimized for ranking and recommendation models. These workloads require frequent AllReduce, AllToAll, and AllGather operations because embedding tables can contain over 99% of model parameters, making communication a bottleneck that competes with computation on GPUs. Its package-integrated network uses two chiplets with twelve custom 800 Gbps RDMA NICs, while 16 message engines and near-memory compute handle collectives independently of the 12×6 processing-element grid. Co-designed HCCL compiles collectives into dependency-aware subgraphs for autonomous execution, with topology-aware algorithms spanning scale-up and scale-out links. In production, HCCL reaches up to 940 GB/s within a rack, and a 150-billion-parameter model across 40 accelerators completed communication 3.9 times faster than an equivalent GPU cluster; concurrent GEMMs degraded less than 0.5%.

Takeaways

1. 1

   Two network chiplets provide twelve custom 800 Gbps RDMA NICs and 1.2 TB/s of total I/O bandwidth without crossing a PCIe bus.

2. 2

   Sixteen message engines independently execute communication, while near-memory compute provides more than 2.8 TB/s of reduction throughput and keeps collectives off the compute grid.

3. 3

   On a 150-billion-parameter production recommendation model using 40 accelerators, MTIA 300’s total communication time was 3.9 times faster than an equivalent GPU cluster; concurrent GEMMs lost less than 0.5% throughput.