Welcome Muse Spark Open: open-weight frontier multimodal mixture-of-experts model from Meta

Summary

Meta released Muse Spark Open, an open-weight multimodal mixture-of-experts model with approximately 816 billion total parameters and around 42 billion active per token. It processes text, images, audio, and video inputs while generating text output, supporting a context window of up to one million tokens. The model uses 256 experts per MoE layer with 8 selected per token, and the BF16 checkpoint requires about 1.5 TB of storage. Meta trained Muse Spark Open for agentic coding tasks including reading codebases, writing and debugging code, and calling tools. The post provides recipes for running the model using transformers, llama.cpp, vLLM, and SGLang, as well as fine-tuning with LoRA on 64 H100 GPUs.

Takeaways

1. 1

   Muse Spark Open has ~816B total parameters but only ~42B active per token due to MoE routing, enabling faster inference while maintaining large knowledge capacity.

2. 2

   The model accepts text, images, audio, and video inputs and uses a single output modality of text, supporting agentic tasks like code generation and browsing.

3. 3

   Fine-tuning Muse Spark Open with LoRA on attention projections is possible on 64 H100s for about 500 steps, achieving a loss drop from 1.26 to 0.57.

FAQ