Topic

Transformer Engine

1 post about Transformer Engine. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Posts

NVIDIA
Efficient MoE Training for Biological Foundation Models
Mixture-of-experts (MoE) architectures scale model capacity more efficiently than dense transformers by activating only a subset of experts per token, but naive implementations suffer from fragmented expert computation, routing overhead, and memory pressure. NVIDIA Transformer Engine (TE) addresses these bottlenecks with optimized primitives including GroupedLinear for batched expert GEMMs, MXFP8 low-precision training, and a fused GroupedMLP kernel that combines quantization, SwiGLU activation, and routing-weight scaling. The BioNeMo recipe implements these techniques for biological foundation models, using TE’s autocast API for MXFP8 precision and the Sequential API to fuse the gate-up, ScaledSwiGLU, and down projections into a single operation. On eight NVIDIA B200 GPUs, the recipe achieved up to 2.21x the training throughput of the Hugging Face baseline for a Mixtral-8x7B model. The tutorial provides prerequisites, code examples, and configuration commands to reproduce the optimizations.

The best engineering reads, once a week.

Topic

1 post about Transformer Engine. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

NVIDIA
Efficient MoE Training for Biological Foundation Models
Mixture-of-experts (MoE) architectures scale model capacity more efficiently than dense transformers by activating only a subset of experts per token, but naive implementations suffer from fragmented expert computation, routing overhead, and memory pressure. NVIDIA Transformer Engine (TE) addresses these bottlenecks with optimized primitives including GroupedLinear for batched expert GEMMs, MXFP8 low-precision training, and a fused GroupedMLP kernel that combines quantization, SwiGLU activation, and routing-weight scaling. The BioNeMo recipe implements these techniques for biological foundation models, using TE’s autocast API for MXFP8 precision and the Sequential API to fuse the gate-up, ScaledSwiGLU, and down projections into a single operation. On eight NVIDIA B200 GPUs, the recipe achieved up to 2.21x the training throughput of the Hugging Face baseline for a Mixtral-8x7B model. The tutorial provides prerequisites, code examples, and configuration commands to reproduce the optimizations.

The best engineering reads, once a week.