---
title: "Transformer Engine"
description: "1 posts about Transformer Engine, summarised, each linking to the original."
---

# Transformer Engine
> 1 posts about Transformer Engine, summarised, each linking to the original.

## Articles

### [Efficient MoE Training for Biological Foundation Models](https://yomu.fyi/post/efficient-moe-training-for-biological-foundation-models.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Michelle Horton
- Published: Sep 24, 2026
- Type: Tutorial

Mixture-of-experts (MoE) architectures scale model capacity more efficiently than dense transformers by activating only a subset of experts per token, but naive implementations suffer from fragmented expert computation, routing overhead, and memory pressure. NVIDIA Transformer Engine (TE) addresses these bottlenecks with optimized primitives including GroupedLinear for batched expert GEMMs, MXFP8 low-precision training, and a fused GroupedMLP kernel that combines quantization, SwiGLU activation, and routing-weight scaling. The BioNeMo recipe implements these techniques for biological foundation models, using TE’s autocast API for MXFP8 precision and the Sequential API to fuse the gate-up, ScaledSwiGLU, and down projections into a single operation. On eight NVIDIA B200 GPUs, the recipe achieved up to 2.21x the training throughput of the Hugging Face baseline for a Mixtral-8x7B model. The tutorial provides prerequisites, code examples, and configuration commands to reproduce the optimizations.