---
title: "Better MoE model inference with warp decode"
description: "MoE inference commonly organizes decode around experts, but the post describes “warp decode,” an output-centric kernel for small-batch generation on NVIDIA Blackwell GPUs. Instead of staging tokens into expert-major buffers, each warp owns one output scalar, streams the required weight rows, accumulates contributions from all eight routed experts in FP32, and uses warp-level reductions; the gate/up and down projections are compressed into two kernels. This removes padding, scatter, combine, activation gather, and per-expert output buffers, while keeping BF16 activations and avoiding intermediate quantization. On B200s, the approach delivered 1.84x higher decode throughput, sustained 3.95 TB/s at batch size 32, and produced outputs 1.4x closer to a full FP32 reference."
---

# Better MoE model inference with warp decode

[Cursor](https://yomu.fyi/company/cursor) · Less Wright, Federico Cassano & Zhiyuan Zhang · Apr 6, 2026

**Type:** Benchmark

## Summary

MoE inference commonly organizes decode around experts, but the post describes “warp decode,” an output-centric kernel for small-batch generation on NVIDIA Blackwell GPUs. Instead of staging tokens into expert-major buffers, each warp owns one output scalar, streams the required weight rows, accumulates contributions from all eight routed experts in FP32, and uses warp-level reductions; the gate/up and down projections are compressed into two kernels. This removes padding, scatter, combine, activation gather, and per-expert output buffers, while keeping BF16 activations and avoiding intermediate quantization. On B200s, the approach delivered 1.84x higher decode throughput, sustained 3.95 TB/s at batch size 32, and produced outputs 1.4x closer to a full FP32 reference.

## Context

Expert-centric MoE execution works well for prefill and large batches, but during autoregressive decode there is too little shared work per expert to amortize the overhead of organizing data. The traditional path includes multiple bookkeeping stages and intermediate buffers that perform no actual computation for a single-token decode.

## Approach / What changed

Warp decode flips the parallelism axis from experts to outputs. Each warp computes one output value, streams the needed weight rows, folds all eight routed experts into a running FP32 accumulator, and writes one result. The gate/up and down projections are implemented as two kernels without staging, cross-warp synchronization, or intermediate buffers.

## Takeaways

- Warp decode assigns each warp a single output scalar, allowing expert contributions and routing weights to be accumulated without materializing eight intermediate results in memory.
- On NVIDIA B200 GPUs, warp decode sustained 3.95 TB/s at batch size 32, equal to 58% of the measured 6.8 TB/s peak for contiguous memory reads.
- Outputs were 1.4x closer to full 32-bit ground truth because BF16 activations remain unquantized and reductions use FP32 accumulators.

**Tags:** [GPU](https://yomu.fyi/topic/gpu), [MXFP8](https://yomu.fyi/topic/mxfp8), [Performance](https://yomu.fyi/topic/performance)

- Source: [Cursor](https://cursor.com/blog/warp-decode)
- Source URL: https://cursor.com/blog/warp-decode
- Ingested by Yomu: 2026-09-02T11:43:00.618Z

[Read original post](https://cursor.com/blog/warp-decode)