Loading…
Topic
12 posts about Mixture of Experts. Every summary links to the original.
See the idea
Blackwell inference is a pipeline, not a chip upgrade. This explainer walks five stages — fleet validation, batching, KV caching, low-precision kernels, and speculative decoding — that turn peak FLOPs into cheap tokens.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Meta launches Muse Spark Open, an 816B-parameter mixture-of-experts model with 42B active parameters and a 1M token context window, trained on NVIDIA GB300 NVL72 for agentic coding workflows. It plans, acts, observes, and self-corrects across multistep tasks like multifile refactoring, using tool calls and subagent coordination while holding an entire project in context without breaking it into pieces. Its 1.8B perception encoder natively processes interleaved text, images, video frames, and audio in a single pass. On GB300 NVL72 with speculative decoding, it achieves over 50K tokens per second per GPU and over 600 tokens per second per user. Sparse MoE routing activates only 42B parameters per token, and NVFP4 quantization reduces memory footprint for larger KV cache. The model is available via Hugging Face and NVIDIA NIM, with support from SGLang, vLLM, and TensorRT LLM for serving, and fine-tuning through NeMo AutoModel and Megatron Bridge.
Red Hat AI released optimized FP8 and DSpark speculative decoding checkpoints for the Nemotron 3.5 Super VL 120B multimodal mixture-of-experts (MoE) model. These checkpoints are designed to double inference throughput on NVIDIA Hopper GPUs without compromising output accuracy. FP8 quantization uses LLM Compressor to reduce model weights and activations from BF16 to 8-bit floating point, cutting memory and storage requirements in half while preserving quality. DSpark speculative decoding proposes 8-16 tokens in a single forward pass and uses a lightweight Markov head to introduce dependencies between speculated tokens. Benchmarks on HumanEval, math_reasoning, and RAG datasets show that each technique delivers meaningful gains individually and combining them outperforms any intermediate configuration at every concurrency level.
NVIDIA released Nemotron 3.5 Super VL, an open 120B-parameter multimodal model for specialized enterprise agents. Its mixture-of-experts architecture activates 12B parameters per token, balancing capacity and inference efficiency. The model combines reasoning, instruction following, tool use, and understanding of text, images, documents, and video for tasks including coding, web research, document intelligence, and cybersecurity. On a single GPU, it achieves up to 5x higher throughput for agentic tasks than competing open models, and it scores higher than the larger Nemotron 3 Ultra on Hermes Bench and LangChain evaluations. Early adopters report cost reductions of 40x for credit processing and 10x lower latency for receipt parsing, and organizations can post-train the model with NVIDIA NeMo for domain-specific workflows and deploy it with NVIDIA NIM and Dynamo, which serves up to 8x more users with 85% lower first-token latency.
Meta released Muse Spark Open, an open-weight multimodal mixture-of-experts model with approximately 816 billion total parameters and around 42 billion active per token. It processes text, images, audio, and video inputs while generating text output, supporting a context window of up to one million tokens. The model uses 256 experts per MoE layer with 8 selected per token, and the BF16 checkpoint requires about 1.5 TB of storage. Meta trained Muse Spark Open for agentic coding tasks including reading codebases, writing and debugging code, and calling tools. The post provides recipes for running the model using transformers, llama.cpp, vLLM, and SGLang, as well as fine-tuning with LoRA on 64 H100 GPUs.
LFM2.5-24B-A2B is a 24-billion-parameter reasoning model with only 2 billion active parameters per token, designed for fast, private on-device agents. It uses a Mixture-of-Experts architecture with 40 layers, 64 experts per block, and top-4 routing, combined with gated short convolutions and GQA. The model produces an explicit chain of thought before answering and supports tool calling and multi-step tasks. Pre-training scaled to 33 trillion tokens, followed by mid-training and a five-stage post-training pipeline including supervised fine-tuning, multi-domain distillation, and agentic reinforcement learning. On benchmarks, it leads or matches larger models on tool use and instruction following, achieving 48.09 on LiveClaw and 96.75 on IFEval, while decoding 184 tokens per second on an Apple M5 Max.
Holo4 is a new series of agentic models designed to interact with software through any available interface: GUIs, code, MCP, and APIs. It comes in two sizes, a 27B dense model and a 35B-A3B Mixture of Experts model, both available on the H Models API. The models were trained through supervised and reinforcement learning on environments and tasks generated by the Agentic Task Factory, which produces verifiable tasks from documentation alone. On the OSWorld 2.0 benchmark, Holo4 27B scores 61.7% against 81.8% for Opus 5.5, achieving competitive performance at a fraction of the cost. The company also released Holotron4 Nano, an updated version of Holotron 3 built on the Nemotron 3 Nano Omni model using the same post-training stack.
Mixture-of-experts (MoE) architectures scale model capacity more efficiently than dense transformers by activating only a subset of experts per token, but naive implementations suffer from fragmented expert computation, routing overhead, and memory pressure. NVIDIA Transformer Engine (TE) addresses these bottlenecks with optimized primitives including GroupedLinear for batched expert GEMMs, MXFP8 low-precision training, and a fused GroupedMLP kernel that combines quantization, SwiGLU activation, and routing-weight scaling. The BioNeMo recipe implements these techniques for biological foundation models, using TE’s autocast API for MXFP8 precision and the Sequential API to fuse the gate-up, ScaledSwiGLU, and down projections into a single operation. On eight NVIDIA B200 GPUs, the recipe achieved up to 2.21x the training throughput of the Hugging Face baseline for a Mixtral-8x7B model. The tutorial provides prerequisites, code examples, and configuration commands to reproduce the optimizations.
Kimi K3 is presented as Moonshot AI’s 2.8-trillion-parameter open-weights model for long-horizon coding, end-to-end knowledge work, and deep reasoning, served through Together AI’s OpenAI-compatible API. Its architecture combines Kimi Delta Attention (KDA), Attention Residuals, and Stable LatentMoE, which activates 16 of 896 experts per token—roughly 2%—and supports a 1M-token context. The implementation guide shows Python calls for configurable reasoning effort, reasoning-free instant mode, streaming, multi-image vision, strict JSON-schema output, and function tools, while noting automatic prefix caching. It also covers production-oriented details such as generous max_tokens for reasoning-heavy structured output, a 100 MB request-body limit for vision, tool-call history handling, and deployment from US infrastructure.
Together AI announces a strategic partnership with Moonshot AI to serve Kimi models as a launch platform, beginning with Kimi K3 and extending to future open-weights releases. Kimi K3 is described as a 2.8-trillion-parameter sparse Mixture-of-Experts model with native vision and a 1M-token context window, using Kimi Delta Attention, Attention Residuals, per-head Muon, and quantile-based expert load balancing. Together says developers can access the models through Serverless, Provisioned Throughput, and Dedicated Inference, with US-hosted infrastructure, zero data retention, autoscaling, and a 99% uptime SLA for Provisioned Throughput. The partnership also supports post-training through full-weight and LoRA reinforcement learning and supervised fine-tuning, with checkpoints deployable directly to inference; Kimi K3 is available at launch, while future Moonshot releases are planned for the same platform.
Moonshot released Kimi K3, a 2.8 trillion parameter multimodal mixture-of-experts transformer with a 1 million token context window and native vision, now on Modal. Modal runs the model at 460 tokens per second on release day through a partnership with Moonshot and vLLM, with token-based pricing on a Shared API or dedicated Auto Endpoint. K3 activates 16 of 896 experts per token and uses Kimi Delta Attention along with Attention Residuals to achieve roughly 2.5 times the scaling efficiency of its predecessor K2. Moonshot employed quantization-aware training from the SFT stage with MXFP4 weights and MXFP8 activations, and contributed a new prefix caching implementation to vLLM to address challenges from KDA. Modal deployed a custom DFlash speculative decoding drafter tuned to K3’s architecture, yielding 360 percent faster interactivity and 88 percent higher throughput on agentic workloads.
Thinking Machines released Inkling, a general-purpose multimodal model that accepts text, image and audio inputs and generates text outputs. The 975B-parameter mixture-of-experts transformer activates 41B parameters per token, uses a 1M-token context window, and employs local attention: five sliding window layers for every global layer. Modal provided day-zero support with a custom DFlash block-diffusion speculator, achieving 250 tokens per second per user on 8x B200s with 67% faster interactivity than the model’s built-in multi-token prediction path. The DFlash drafter uses all local attention and causal layers, keeping cost flat as block size grows. Inkling is available today as an OpenAI-compatible shared endpoint through Modal Auto Endpoints with token-based pricing, allowing users to try the model interactively.
Thinking Machines Lab released Inkling, and Together AI announced day-zero access through its production inference platform. The multimodal mixture-of-experts model accepts text, image, and audio inputs and generates text through a unified decoder architecture, with controllable inference effort for trading reasoning depth, token usage, and latency. Inkling has 975B total parameters, 40B active parameters per token, a 1M-token context window, query-conditioned relative attention, short causal convolutions, and shared-sink MoE routing; Together serves it with a FlashAttention-4-based kernel. Preliminary evaluations report strong results across scientific reasoning, mathematics, coding, agentic, vision, and audio benchmarks, while the model is also post-trained for forecasting and calibrated prediction. It is available on Together Serverless through OpenAI-compatible APIs without infrastructure provisioning.
The best engineering reads, once a week.
