---
title: "MoE"
description: "1 posts about MoE, summarised, each linking to the original."
---

# MoE
> 1 posts about MoE, summarised, each linking to the original.

## Articles

### [Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs](https://yomu.fyi/post/introducing-olmo-core-3-open-scalable-training-infrastructure-for-larg.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Kyle Wiggers
- Published: Oct 1, 2026
- Type: Announcement

Olmo-core 3 is an open training infrastructure upgrade for large mixture-of-experts (MoE) models, designed to scale to trillion parameters while preserving efficiency. It switches from FSDP to a DDP-based system that keeps experts resident on GPUs and routes data to them, avoiding weight gathering overhead. The system combines expert, pipeline, and distributed optimizer parallelism with optimizations like rowwise expert parallelism, GPU-resident routing, and grouped GEMM. In a benchmark, increasing experts from 8 to 128 while routing each token to four experts kept active parameters at ~3.2B, total parameters grew from 4.6B to 47B with throughput falling less than 5%. The infrastructure supports MXFP8 lower-precision format, achieving about 21% higher throughput than BF16, and has been tested on a 1.2-trillion-parameter model.