--- title: "Mixture of Experts" description: "13 posts about Mixture of Experts, summarised, each linking to the original." --- # Mixture of Experts > 13 posts about Mixture of Experts, summarised, each linking to the original. ## Articles ### [Welcome Beam: a 501B open model you can own, customize, and scale](https://yomu.fyi/post/welcome-beam-a-501b-open-model-you-can-own-customize-and-scale.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: ben burtenshaw, merve, Pedro Cuenca - Published: Oct 16, 2026 - Type: Announcement Reflection released Beam, a 501B-parameter mixture-of-experts model with 23B active parameters per token, under the Apache 2.0 license, targeting coding and agentic use cases. The model supports a 1M-token context window using hybrid attention with 43 sliding-window and 9 global layers, and a residual stream that scales sublayer outputs by 1/sqrt(k) to maintain constant magnitude. Its MoE router uses a group-limited scheme with 256 experts, 8 routed per token, and the BF16 checkpoint occupies about 1 TB. Benchmarks show strong performance: 81.6% on SWE-bench Verified, 97.7% on AIME 2026, and 90.6% on GPQA Diamond, with minimal degradation in FP8 and NVFP4 quantizations. The model ships with a Rust-based harness, day-0 support in transformers, vLLM, and SGLang, and includes SFT and RL recipes, with a $10,000 Hub credits competition for community projects. ### [Scale Agentic Coding Workflows with Meta’s Muse Spark Open on NVIDIA GB300 NVL72](https://yomu.fyi/post/scale-agentic-coding-workflows-with-meta-s-muse-spark-open-on-nvidia-g.md) - Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md) - Author: Tanya Lenz - Published: Oct 15, 2026 - Type: Announcement Meta launches Muse Spark Open, an 816B-parameter mixture-of-experts model with 42B active parameters and a 1M token context window, trained on NVIDIA GB300 NVL72 for agentic coding workflows. It plans, acts, observes, and self-corrects across multistep tasks like multifile refactoring, using tool calls and subagent coordination while holding an entire project in context without breaking it into pieces. Its 1.8B perception encoder natively processes interleaved text, images, video frames, and audio in a single pass. On GB300 NVL72 with speculative decoding, it achieves over 50K tokens per second per GPU and over 600 tokens per second per user. Sparse MoE routing activates only 42B parameters per token, and NVFP4 quantization reduces memory footprint for larger KV cache. The model is available via Hugging Face and NVIDIA NIM, with support from SGLang, vLLM, and TensorRT LLM for serving, and fine-tuning through NeMo AutoModel and Megatron Bridge. ### [Optimized support for Open NVIDIA Nemotron 3.5 Super VL](https://yomu.fyi/post/optimized-support-for-open-nvidia-nemotron-3-5-super-vl.md) - Company: [Red Hat](https://yomu.fyi/company/red-hat.md) - Author: Eldar Kurtić, Helen Zhao, Charles Hernandez, Dipika Sikka - Published: Oct 15, 2026 - Type: Announcement Red Hat AI released optimized FP8 and DSpark speculative decoding checkpoints for the Nemotron 3.5 Super VL 120B multimodal mixture-of-experts (MoE) model. These checkpoints are designed to double inference throughput on NVIDIA Hopper GPUs without compromising output accuracy. FP8 quantization uses LLM Compressor to reduce model weights and activations from BF16 to 8-bit floating point, cutting memory and storage requirements in half while preserving quality. DSpark speculative decoding proposes 8-16 tokens in a single forward pass and uses a lightweight Markov head to introduce dependencies between speculated tokens. Benchmarks on HumanEval, math\_reasoning, and RAG datasets show that each technique delivers meaningful gains individually and combining them outperforms any intermediate configuration at every concurrency level. ### [NVIDIA Nemotron 3.5 Super VL Powers Specialized Agents with Multimodal Reasoning](https://yomu.fyi/post/nvidia-nemotron-3-5-super-vl-powers-specialized-agents-with-multimodal.md) - Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md) - Author: Michelle Horton - Published: Oct 15, 2026 - Type: Announcement NVIDIA released Nemotron 3.5 Super VL, an open 120B-parameter multimodal model for specialized enterprise agents. Its mixture-of-experts architecture activates 12B parameters per token, balancing capacity and inference efficiency. The model combines reasoning, instruction following, tool use, and understanding of text, images, documents, and video for tasks including coding, web research, document intelligence, and cybersecurity. On a single GPU, it achieves up to 5x higher throughput for agentic tasks than competing open models, and it scores higher than the larger Nemotron 3 Ultra on Hermes Bench and LangChain evaluations. Early adopters report cost reductions of 40x for credit processing and 10x lower latency for receipt parsing, and organizations can post-train the model with NVIDIA NeMo for domain-specific workflows and deploy it with NVIDIA NIM and Dynamo, which serves up to 8x more users with 85% lower first-token latency. ### [Welcome Muse Spark Open: open-weight frontier multimodal mixture-of-experts model from Meta](https://yomu.fyi/post/welcome-muse-spark-open-open-weight-frontier-multimodal-mixture-of-exp.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Pedro Cuenca, merve, ben burtenshaw, Guarin Flück, Quentin Gallouédec, Alvaro Bartolome, Serveurperso, Jun Kim - Published: Oct 15, 2026 - Type: Announcement Meta released Muse Spark Open, an open-weight multimodal mixture-of-experts model with approximately 816 billion total parameters and around 42 billion active per token. It processes text, images, audio, and video inputs while generating text output, supporting a context window of up to one million tokens. The model uses 256 experts per MoE layer with 8 selected per token, and the BF16 checkpoint requires about 1.5 TB of storage. Meta trained Muse Spark Open for agentic coding tasks including reading codebases, writing and debugging code, and calling tools. The post provides recipes for running the model using transformers, llama.cpp, vLLM, and SGLang, as well as fine-tuning with LoRA on 64 H100 GPUs. ### [Run fast, private agents on-device with LFM2.5-24B-A2B](https://yomu.fyi/post/run-fast-private-agents-on-device-with-lfm2-5-24b-a2b.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Raahul Vignesh, Song Duong, Akhilesh Deepak Gotmare, Fernando Fernandes Neto, Jiahui Wang, Mayank Agarwal, nathan ranchin, Sergei Tilga, Sinoué GAD, Tim Seyde, Zetian Li, changyi yang, Simon Lee, Leonie Monigatti, Maxime Labonne, Karthik Macherla, Alfred Eng, Khoi Nguyen, Paul Pak, xx, Tarek Dakhran, Anthony Maltsev - Published: Oct 13, 2026 - Type: Announcement LFM2.5-24B-A2B is a 24-billion-parameter reasoning model with only 2 billion active parameters per token, designed for fast, private on-device agents. It uses a Mixture-of-Experts architecture with 40 layers, 64 experts per block, and top-4 routing, combined with gated short convolutions and GQA. The model produces an explicit chain of thought before answering and supports tool calling and multi-step tasks. Pre-training scaled to 33 trillion tokens, followed by mid-training and a five-stage post-training pipeline including supervised fine-tuning, multi-domain distillation, and agentic reinforcement learning. On benchmarks, it leads or matches larger models on tool use and instruction following, achieving 48.09 on LiveClaw and 96.75 on IFEval, while decoding 184 tokens per second on an Apple M5 Max. ### [Holo4: powering generalist computer-use agents](https://yomu.fyi/post/holo4-powering-generalist-computer-use-agents.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Tony Wu, Maxime Theillard, Frederic Renard, Vincent Coyette, Emrick Sinitambirivoutin, Avshalom Manevich, Antonio Loison, Antoine Bonnet, Maxime Langevin, Aleix Cambray (H-AI), Léonard Benedetti, Tony Wu, Mats L. Richter, Michael Eickenberg, Sławek Mucha, Matthias Brunel, Daniel Beechey - Published: Sep 28, 2026 - Type: Announcement Holo4 is a new series of agentic models designed to interact with software through any available interface: GUIs, code, MCP, and APIs. It comes in two sizes, a 27B dense model and a 35B-A3B Mixture of Experts model, both available on the H Models API. The models were trained through supervised and reinforcement learning on environments and tasks generated by the Agentic Task Factory, which produces verifiable tasks from documentation alone. On the OSWorld 2.0 benchmark, Holo4 27B scores 61.7% against 81.8% for Opus 5.5, achieving competitive performance at a fraction of the cost. The company also released Holotron4 Nano, an updated version of Holotron 3 built on the Nemotron 3 Nano Omni model using the same post-training stack. ### [Efficient MoE Training for Biological Foundation Models](https://yomu.fyi/post/efficient-moe-training-for-biological-foundation-models.md) - Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md) - Author: Michelle Horton - Published: Sep 24, 2026 - Type: Tutorial Mixture-of-experts (MoE) architectures scale model capacity more efficiently than dense transformers by activating only a subset of experts per token, but naive implementations suffer from fragmented expert computation, routing overhead, and memory pressure. NVIDIA Transformer Engine (TE) addresses these bottlenecks with optimized primitives including GroupedLinear for batched expert GEMMs, MXFP8 low-precision training, and a fused GroupedMLP kernel that combines quantization, SwiGLU activation, and routing-weight scaling. The BioNeMo recipe implements these techniques for biological foundation models, using TE’s autocast API for MXFP8 precision and the Sequential API to fuse the gate-up, ScaledSwiGLU, and down projections into a single operation. On eight NVIDIA B200 GPUs, the recipe achieved up to 2.21x the training throughput of the Hugging Face baseline for a Mixtral-8x7B model. The tutorial provides prerequisites, code examples, and configuration commands to reproduce the optimizations. ### [Kimi K3: the complete developer guide](https://yomu.fyi/post/kimi-k3-the-complete-developer-guide.md) - Company: [Together AI](https://yomu.fyi/company/together-ai.md) - Published: Aug 1, 2026 - Type: Tutorial Kimi K3 is presented as Moonshot AI’s 2.8-trillion-parameter open-weights model for long-horizon coding, end-to-end knowledge work, and deep reasoning, served through Together AI’s OpenAI-compatible API. Its architecture combines Kimi Delta Attention (KDA), Attention Residuals, and Stable LatentMoE, which activates 16 of 896 experts per token—roughly 2%—and supports a 1M-token context. The implementation guide shows Python calls for configurable reasoning effort, reasoning-free instant mode, streaming, multi-image vision, strict JSON-schema output, and function tools, while noting automatic prefix caching. It also covers production-oriented details such as generous max\_tokens for reasoning-heavy structured output, a 100 MB request-body limit for vision, tool-call history handling, and deployment from US infrastructure. ### [Together AI announces strategic partnership with Moonshot AI to natively serve Kimi models](https://yomu.fyi/post/together-ai-announces-strategic-partnership-with-moonshot-ai-to-native.md) - Company: [Together AI](https://yomu.fyi/company/together-ai.md) - Published: Jul 29, 2026 - Type: Announcement Together AI announces a strategic partnership with Moonshot AI to serve Kimi models as a launch platform, beginning with Kimi K3 and extending to future open-weights releases. Kimi K3 is described as a 2.8-trillion-parameter sparse Mixture-of-Experts model with native vision and a 1M-token context window, using Kimi Delta Attention, Attention Residuals, per-head Muon, and quantile-based expert load balancing. Together says developers can access the models through Serverless, Provisioned Throughput, and Dedicated Inference, with US-hosted infrastructure, zero data retention, autoscaling, and a 99% uptime SLA for Provisioned Throughput. The partnership also supports post-training through full-weight and LoRA reinforcement learning and supervised fine-tuning, with checkpoints deployable directly to inference; Kimi K3 is available at launch, while future Moonshot releases are planned for the same platform. ### [Kimi K3 by Moonshot now available on Modal](https://yomu.fyi/post/kimi-k3-by-moonshot-now-available-on-modal.md) - Company: [Modal](https://yomu.fyi/company/modal.md) - Published: Jul 27, 2026 - Type: Announcement Moonshot released Kimi K3, a 2.8 trillion parameter multimodal mixture-of-experts transformer with a 1 million token context window and native vision, now on Modal. Modal runs the model at 460 tokens per second on release day through a partnership with Moonshot and vLLM, with token-based pricing on a Shared API or dedicated Auto Endpoint. K3 activates 16 of 896 experts per token and uses Kimi Delta Attention along with Attention Residuals to achieve roughly 2.5 times the scaling efficiency of its predecessor K2. Moonshot employed quantization-aware training from the SFT stage with MXFP4 weights and MXFP8 activations, and contributed a new prefix caching implementation to vLLM to address challenges from KDA. Modal deployed a custom DFlash speculative decoding drafter tuned to K3’s architecture, yielding 360 percent faster interactivity and 88 percent higher throughput on agentic workloads. ### [Inkling by Thinking Machines now available on Modal](https://yomu.fyi/post/inkling-by-thinking-machines-now-available-on-modal.md) - Company: [Modal](https://yomu.fyi/company/modal.md) - Published: Jul 15, 2026 - Type: Announcement Thinking Machines released Inkling, a general-purpose multimodal model that accepts text, image and audio inputs and generates text outputs. The 975B-parameter mixture-of-experts transformer activates 41B parameters per token, uses a 1M-token context window, and employs local attention: five sliding window layers for every global layer. Modal provided day-zero support with a custom DFlash block-diffusion speculator, achieving 250 tokens per second per user on 8x B200s with 67% faster interactivity than the model’s built-in multi-token prediction path. The DFlash drafter uses all local attention and causal layers, keeping cost flat as block size grows. Inkling is available today as an OpenAI-compatible shared endpoint through Modal Auto Endpoints with token-based pricing, allowing users to try the model interactively. ### [Together AI brings Thinking Machines Lab’s new model Inkling on day 0](https://yomu.fyi/post/together-ai-brings-thinking-machines-lab-s-new-model-inkling-on-day-0.md) - Company: [Together AI](https://yomu.fyi/company/together-ai.md) - Published: Jul 15, 2026 - Type: Announcement Thinking Machines Lab released Inkling, and Together AI announced day-zero access through its production inference platform. The multimodal mixture-of-experts model accepts text, image, and audio inputs and generates text through a unified decoder architecture, with controllable inference effort for trading reasoning depth, token usage, and latency. Inkling has 975B total parameters, 40B active parameters per token, a 1M-token context window, query-conditioned relative attention, short causal convolutions, and shared-sink MoE routing; Together serves it with a FlashAttention-4-based kernel. Preliminary evaluations report strong results across scientific reasoning, mathematics, coding, agentic, vision, and audio benchmarks, while the model is also post-trained for forecasting and calibrated prediction. It is available on Together Serverless through OpenAI-compatible APIs without infrastructure provisioning.