The Design Space of Tri-Modal Masked Diffusion Models

Summary

Discrete diffusion models have emerged as strong alternatives to autoregressive language models, but prior approaches typically initialize a unimodal model and fine-tune for bimodal generation. This work diverges from that paradigm by introducing the first tri-modal Masked Diffusion Model pretrained from scratch jointly on text, image-text, and audio-text data, and systematically analyzing multimodal scaling laws, modality mixing ratios, noise schedules, and batch-size effects. A key contribution is a stochastic differential equation reparameterization that decouples physical and logical batch sizes, removing the need for batch size tuning reported in recent work. The authors pretrain a 3B-parameter model on 6.4 trillion tokens and provide optimized inference sampling defaults, achieving strong results on text generation, text-to-image, and text-to-speech tasks. This work represents the largest open systematic study of multimodal discrete diffusion models conducted to date.

Takeaways

1. 1

   This work presents the first tri-modal Masked Diffusion Model pretrained from scratch on text, image-text, and audio-text data.

2. 2

   A novel SDE-based reparameterization decouples physical batch size from logical batch size, eliminating the need for optimal batch size tuning.

3. 3

   The 3B-parameter model trained on 6.4T tokens achieves strong results on text generation, text-to-image, and text-to-speech tasks.

FAQ