---
title: "Mixture of Experts (MoEs) in Transformers"
description: "Mixture of Experts (MoEs) in Transformers let a Transformer retain its backbone while replacing selected feed-forward layers with routed expert sub-networks, allowing total model capacity to exceed the active parameters used per token. The post explains why sparse models can reduce inference cost and memory pressure, then focuses on making them first-class in Transformers. Its weight-loading refactor converts independently serialized expert tensors into packed runtime layouts through composable WeightConverter operations and lazy materialization. The Experts Backend supplies eager, batched_mm, and grouped_mm execution strategies, while expert parallelism shards experts across devices and aggregates partial outputs. Training support uses grouped GEMM and custom Triton plus LoRA kernels, with reported improvements of approximately 12× speed, over 35% VRAM reduction, and roughly 6× longer context."
---

# Mixture of Experts (MoEs) in Transformers

[Hugging Face](https://yomu.fyi/company/hugging-face) · Aritra Roy Gosthipaty, Pedro Cuenca, merve, Ilyas Moutawwakil, Arthur Zucker, Sergio Paniego, Pablo Montalvo · Feb 26, 2026

**Type:** Explainer

## Summary

Mixture of Experts (MoEs) in Transformers let a Transformer retain its backbone while replacing selected feed-forward layers with routed expert sub-networks, allowing total model capacity to exceed the active parameters used per token. The post explains why sparse models can reduce inference cost and memory pressure, then focuses on making them first-class in Transformers. Its weight-loading refactor converts independently serialized expert tensors into packed runtime layouts through composable WeightConverter operations and lazy materialization. The Experts Backend supplies eager, batched\_mm, and grouped\_mm execution strategies, while expert parallelism shards experts across devices and aggregates partial outputs. Training support uses grouped GEMM and custom Triton plus LoRA kernels, with reported improvements of approximately 12× speed, over 35% VRAM reduction, and roughly 6× longer context.

## Context

Dense language-model scaling increases training expense, inference latency, and deployment memory requirements. MoE models address this by using a router to select only a small subset of learnable experts for each token, while retaining a larger total parameter capacity. Supporting these sparse architectures also requires changes because existing tooling was primarily designed for dense models, and MoE training involves large parameter counts, distributed expert communication, and routing instabilities.

## Approach / What changed

Transformers introduces a weight-loading refactor centered on WeightConverter, which maps serialized expert tensors into packed runtime layouts using composable operations such as MergeModulelist, Concatenate, and SplitModulelist. Lazy materialization schedules conversions after dependencies are ready. The Experts Backend decouples expert execution from model implementations and provides eager, batched\_mm, and grouped\_mm backends. Expert parallelism distributes experts across devices, while training optimizations use grouped GEMM, Triton grouped-GEMM, and LoRA kernels.

## Takeaways

- WeightConverter treats checkpoint loading as a tensor-conversion pipeline, allowing independently serialized expert weights to be merged into contiguous tensors required by optimized MoE kernels.
- The Experts Backend offers three execution strategies: eager for reference and debugging, batched\_mm for small batch GPU-heavy workloads, and grouped\_mm for larger batches or memory-constrained setups.
- Expert parallelism loads each device's assigned experts and uses routing plus all-reduce aggregation; the documented training work reports approximately 12× faster MoE training, over 35% lower VRAM use, and roughly 6× longer context.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Performance](https://yomu.fyi/topic/performance)

- Source: [Hugging Face](https://huggingface.co/blog/moe-transformers)
- Source URL: https://huggingface.co/blog/moe-transformers
- Ingested by Yomu: 2026-08-28T20:00:58.242Z

[Read original post](https://huggingface.co/blog/moe-transformers)