Topic

MoE

1 post about MoE. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Posts

Hugging Face
Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs
Olmo-core 3 is an open training infrastructure upgrade for large mixture-of-experts (MoE) models, designed to scale to trillion parameters while preserving efficiency. It switches from FSDP to a DDP-based system that keeps experts resident on GPUs and routes data to them, avoiding weight gathering overhead. The system combines expert, pipeline, and distributed optimizer parallelism with optimizations like rowwise expert parallelism, GPU-resident routing, and grouped GEMM. In a benchmark, increasing experts from 8 to 128 while routing each token to four experts kept active parameters at ~3.2B, total parameters grew from 4.6B to 47B with throughput falling less than 5%. The infrastructure supports MXFP8 lower-precision format, achieving about 21% higher throughput than BF16, and has been tested on a 1.2-trillion-parameter model.

The best engineering reads, once a week.

Topic

1 post about MoE. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Hugging Face
Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs
Olmo-core 3 is an open training infrastructure upgrade for large mixture-of-experts (MoE) models, designed to scale to trillion parameters while preserving efficiency. It switches from FSDP to a DDP-based system that keeps experts resident on GPUs and routes data to them, avoiding weight gathering overhead. The system combines expert, pipeline, and distributed optimizer parallelism with optimizations like rowwise expert parallelism, GPU-resident routing, and grouped GEMM. In a benchmark, increasing experts from 8 to 128 while routing each token to four experts kept active parameters at ~3.2B, total parameters grew from 4.6B to 47B with throughput falling less than 5%. The infrastructure supports MXFP8 lower-precision format, achieving about 21% higher throughput than BF16, and has been tested on a 1.2-trillion-parameter model.

The best engineering reads, once a week.