Loading post…
Hugging FaceKyle WiggersAnnouncement
Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs
Summary
Olmo-core 3 is an open training infrastructure upgrade for large mixture-of-experts (MoE) models, designed to scale to trillion parameters while preserving efficiency. It switches from FSDP to a DDP-based system that keeps experts resident on GPUs and routes data to them, avoiding weight gathering overhead. The system combines expert, pipeline, and distributed optimizer parallelism with optimizations like rowwise expert parallelism, GPU-resident routing, and grouped GEMM. In a benchmark, increasing experts from 8 to 128 while routing each token to four experts kept active parameters at ~3.2B, total parameters grew from 4.6B to 47B with throughput falling less than 5%. The infrastructure supports MXFP8 lower-precision format, achieving about 21% higher throughput than BF16, and has been tested on a 1.2-trillion-parameter model.
Context
Training large AI models is computationally expensive; MoE models improve efficiency by activating only a subset of parameters per token, but the communication overhead of routing data to experts across a cluster can erode this advantage. Olmo-core 3 addresses this gap with a redesigned training infrastructure.
Approach / What changed
Olmo-core 3 adopts a DDP-based training system that keeps MoE experts resident on GPUs and routes data to them, combined with expert, pipeline, and distributed optimizer parallelism, along with optimizations such as rowwise expert parallelism for efficient data routing, GPU-resident routing to avoid CPU-GPU transfers, grouped GEMM for efficient small computations, and MXFP8 lower-precision format for reduced communication and computation.
Takeaways
Olmo-core 3 switches from FSDP to a DDP-based system, improving throughput 2.7x over the previous implementation on a 47B-parameter MoE model.
The training infrastructure combines expert, pipeline, and distributed optimizer parallelism, along with rowwise expert parallelism and GPU-resident routing, to scale MoE models efficiently.
Enabling MXFP8 lower-precision format increased training throughput by about 21% compared to BF16 on a controlled benchmark, while reducing peak memory usage.
FAQ
How does Olmo-core 3 differ from the previous MoE implementation?
The previous implementation used FSDP which gathered and resharded model weights per batch. Olmo-core 3 uses DDP to keep experts resident on GPUs and route data to them, avoiding repeated weight gathering. In a test, this improved throughput by about 2.7x.
What techniques does Olmo-core 3 use to scale MoE training?
It combines expert parallelism (spreading experts across GPUs), pipeline parallelism (splitting layers), and a distributed optimizer. Optimizations include rowwise expert parallelism for efficient data routing, GPU-resident routing to keep metadata on GPUs, and grouped GEMM to combine small expert computations.
How much does MXFP8 improve training throughput?
In a controlled benchmark on four NVIDIA B300 GPUs, enabling MXFP8 across the most beneficial parts increased training throughput about 21% over BF16, while peak memory fell from 103 GiB to 95 GiB.
Source and freshness
- Source
- huggingface.co
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.