---
title: "The Design Space of Tri-Modal Masked Diffusion Models"
description: "Discrete diffusion models have emerged as strong alternatives to autoregressive language models, but prior approaches typically initialize a unimodal model and fine-tune for bimodal generation. This work diverges from that paradigm by introducing the first tri-modal Masked Diffusion Model pretrained from scratch jointly on text, image-text, and audio-text data, and systematically analyzing multimodal scaling laws, modality mixing ratios, noise schedules, and batch-size effects. A key contribution is a stochastic differential equation reparameterization that decouples physical and logical batch sizes, removing the need for batch size tuning reported in recent work. The authors pretrain a 3B-parameter model on 6.4 trillion tokens and provide optimized inference sampling defaults, achieving strong results on text generation, text-to-image, and text-to-speech tasks. This work represents the largest open systematic study of multimodal discrete diffusion models conducted to date."
---

# The Design Space of Tri-Modal Masked Diffusion Models

[Apple](https://yomu.fyi/company/apple) · Oct 14, 2026

**Type:** Announcement

## Summary

Discrete diffusion models have emerged as strong alternatives to autoregressive language models, but prior approaches typically initialize a unimodal model and fine-tune for bimodal generation. This work diverges from that paradigm by introducing the first tri-modal Masked Diffusion Model pretrained from scratch jointly on text, image-text, and audio-text data, and systematically analyzing multimodal scaling laws, modality mixing ratios, noise schedules, and batch-size effects. A key contribution is a stochastic differential equation reparameterization that decouples physical and logical batch sizes, removing the need for batch size tuning reported in recent work. The authors pretrain a 3B-parameter model on 6.4 trillion tokens and provide optimized inference sampling defaults, achieving strong results on text generation, text-to-image, and text-to-speech tasks. This work represents the largest open systematic study of multimodal discrete diffusion models conducted to date.

## Context

Discrete diffusion models have emerged as alternatives to autoregressive language models, but prior work typically initializes a unimodal model and fine-tunes for bimodal generation. This work introduces the first tri-modal Masked Diffusion Model pretrained from scratch on text, image-text, and audio-text data.

## Approach / What changed

The authors systematically analyze multimodal scaling laws, modality mixing ratios, noise schedules, and batch-size effects. They derive a stochastic differential equation based reparameterization that decouples physical and logical batch sizes. They pretrain a 3B-parameter model on 6.4 trillion tokens and provide optimized inference sampling defaults.

## Takeaways

- This work presents the first tri-modal Masked Diffusion Model pretrained from scratch on text, image-text, and audio-text data.
- A novel SDE-based reparameterization decouples physical batch size from logical batch size, eliminating the need for optimal batch size tuning.
- The 3B-parameter model trained on 6.4T tokens achieves strong results on text generation, text-to-image, and text-to-speech tasks.

## FAQ

### How does this tri-modal model differ from prior multimodal diffusion models?

Prior approaches typically initialize from a unimodal model and fine-tune for bimodal generation. This model is the first pretrained from scratch on text, image-text, and audio-text data jointly, enabling tri-modal capabilities.

### What problem does the SDE reparameterization solve?

It decouples the physical batch size, chosen for compute efficiency (GPU saturation, wall clock), from the logical batch size, chosen to balance gradient variance. This removes the need for tuning optimal batch size as reported in recent work.

### What are the model's capabilities and scale?

The model has 3B parameters and is trained on 6.4 trillion tokens. It achieves strong results on text generation, text-to-image (T2I), and text-to-speech (T2S) tasks.

**Tags:** [Batch Size](https://yomu.fyi/topic/batch-size), [Masked Diffusion](https://yomu.fyi/topic/masked-diffusion), [Multimodal](https://yomu.fyi/topic/multimodal)

- Source: [Apple](https://machinelearning.apple.com/research/tri-modal-masked-diffusion)
- Source URL: https://machinelearning.apple.com/research/tri-modal-masked-diffusion
- Ingested by Yomu: 2026-10-14T18:00:14.662Z

[Read original post](https://machinelearning.apple.com/research/tri-modal-masked-diffusion)