GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model

Summary

Meta’s Generative Ads Recommendation Model (GEM), a hybrid foundation model for ads recommendations on Instagram and Facebook, trains across several thousand latest-generation GPUs and combines trillions of sparse embedding parameters with billions of dense parameters. The post frames its engineering problem around recommendation-specific jagged and asymmetric sequences, memory-bound operations, numerical sensitivity, communication overhead, activation pressure, and load skew that make conventional LLM training techniques insufficient. Its co-design approach combines custom kernels such as Jagged Flash Attention (JFA), BlockAttention, and Generalized Dot-Product Attention (GDPA), mixed ultra-low-precision training including MXFP8 attention and MLP, topology-aware 5D parallelism, SM-free collectives, and memory and load-balancing techniques. The reported result is a doubling of end-to-end training efficiency to 20–25% Model FLOPs Utilization (MFU) while total training FLOPs increased fourfold over 12 months; Base Batch Shuffling (BBS) alone delivered a reported 4% efficiency gain, with 4% QPS improvement and 4% peak memory reduction.

Takeaways

1. 1

   Jagged Flash Attention eliminates compute waste from padding variable-length inputs, while BlockAttention reduces long user-history self-attention cost from O(L²) to O(L) while preserving model quality and efficiency.

2. 2

   Base Batch Shuffling sorts 128-sample sub-batches by total sequence length and interleaves heavy and light groups into 1K+ sample batches, delivering a reported 4% efficiency gain without cross-rank communication.

3. 3

   GEM reached 20–25% end-to-end MFU after a twofold efficiency improvement, while its total training FLOPs scaled fourfold over 12 months.