Introducing Modular Diffusers - Composable Building Blocks for Diffusion Pipelines

Summary

Modular Diffusers introduces a composable alternative to Diffusers’ existing DiffusionPipeline for building diffusion workflows from reusable blocks. A ModularPipeline can load a workflow definition, load model components separately with settings such as bfloat16, and preserve the familiar inference API, while blocks for text encoding, image encoding, denoising, and decoding can be inspected, run independently, or rearranged. Custom Python blocks declare components, inputs, outputs, and computation logic; the example adds a Depth Anything V2 depth processor to a Qwen Image ControlNet workflow, with data flowing between compatible blocks. Blocks and complete pipelines can be published to the Hub, and Mellon can generate node interfaces from their consistent definitions. The post presents the system as an early, flexible direction and says Mellon is not ready for production use.

Takeaways

1. 1

   A ModularPipeline separates workflow definition from component loading: from_pretrained defines the workflow, while load_components loads model weights and supports settings such as torch_dtype and quantization.

2. 2

   Custom blocks declare expected components, inputs, intermediate outputs, and __call__ computation. In the example, a Depth Anything V2 block supplies control_image data to a Qwen Image ControlNet workflow.

3. 3

   Mellon can load Hub-hosted custom blocks and generate their node interfaces from the block definition, but the integration is described as early development and not ready for production use.