Loading post…
NVIDIAElizabeth GoodmanExplainer
Dense vs. MoE Models: Active Parameters, Throughput, and When to Choose Each
Summary
Dense models activate every parameter for every token, while Mixture-of-Experts (MoE) models route each token through a selected subset of feed-forward experts. This changes the relationship between capacity, memory, and compute: total parameters determine the memory required to load a model, whereas active parameters largely determine per-token work. At equal total size, MoE models often provide higher throughput, especially at batch size one, but routing, simultaneous expert residency, and reduced KV-cache room add serving complexity; latency advantages narrow at high concurrency. The comparison uses Nemotron 3.5 Lightning, a roughly 30B-parameter Mamba-2 plus MoE plus Attention hybrid with 3B active parameters, alongside dense and MoE examples, and concludes that dense models suit simpler serving and fine-tuning while MoE models suit throughput-oriented deployments with manageable memory and operational complexity.
Context
The choice between dense and MoE architectures is presented as a deployment decision involving throughput, memory cost, latency, serving complexity, fine-tuning, and quantization. The source compares models with similar total parameter counts and emphasizes that active parameters alone do not describe memory requirements or overall capability.
Approach / What changed
The source explains dense execution and MoE routing at the decoder-layer level, then compares their compute, memory, throughput, latency, fine-tuning, and quantization trade-offs. It uses Nemotron 3.5 Lightning and tabulated model data to connect the architectural differences to deployment choices.
Takeaways
MoE routing occurs separately at every decoder layer, with a learned router selecting top-k experts; tokens still pass through the full attention mechanism.
MoE models keep all experts in GPU memory, so total parameters still determine the memory footprint and can leave less room for KV cache than a comparable dense model.
Dense models are simpler to serve and fine-tune, while MoE models shift cost toward fixed memory and can improve throughput when serving complexity is acceptable.
FAQ
How can a 30B-parameter MoE model activate only 3B parameters per token?
A router selects a small subset of feed-forward expert blocks at each decoder layer, while the remaining experts are skipped for that token. The active-parameter figure still includes the attention and embedding weights used by every token.
Does an MoE model need memory for all of its experts?
Yes. All experts must reside in GPU memory simultaneously, so memory usage tracks total parameters rather than active parameters. This can leave less room for the KV cache than a dense model of comparable size.
When might a dense model be preferable despite lower throughput?
Dense models offer simpler serving, more predictable latency, and simpler full fine-tuning because all parameters activate uniformly. They can be preferable when high-concurrency latency, operational simplicity, or capability per parameter matters more than maximum throughput.
Source and freshness
- Source
- developer.nvidia.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.