# GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model

[Meta](https://yomu.fyi/company/meta) · Aug 3, 2026

## Summary

Meta's Generative Ads Recommendation Model (GEM) trains across thousands of GPUs using a hybrid architecture featuring trillions of sparse embedding parameters and billions of dense parameters. Standard training infrastructure built for large language models struggles with recommendation-specific workloads characterized by jagged user activity histories, asymmetric attention patterns, and numerical sensitivity. To overcome these bottlenecks, engineering teams introduced custom kernels like Jagged Flash Attention and Generalized Dot-Product Attention alongside MXFP8 mixed-precision execution. Pairing these kernel optimizations with topology-aware 5D parallelism doubled end-to-end training efficiency to 20–25% Model FLOPs Utilization while scaling total training FLOPs fourfold over twelve months.

## Takeaways

- Padding variable-length user activity sequences can waste up to 50% of compute, which Jagged Flash Attention avoids by executing variable-length tensors directly using Tensor Memory Accelerator subtraction schemes.
- End-to-end Model FLOPs Utilization (MFU) is decoupled into single-GPU compute efficiency (Local MFU) and multi-GPU distributed retention (Scaling Ratio) to address kernel design and distributed communication as separate problems.
- GEM scales dense parameters using 2D Fully Sharded Data Parallelism combined with Expert Parallelism, while sparse embedding parameters are distributed using Fully Sharded 2D Model Parallelism.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance), [Recommendation Systems](https://yomu.fyi/topic/recommendation-systems), [Scalability](https://yomu.fyi/topic/scalability)

[Read original post](https://engineering.fb.com/2026/08/03/ml-applications/training-gem-at-llm-scale-meta-ads-recommendation-foundation-model)