Diffusers welcomes FLUX-2

Summary

Diffusers introduces support for FLUX-2 models through the Flux2Pipeline and Flux2Transformer2DModel classes. Running the 4-bit quantized checkpoint diffusers/FLUX.2-dev-bnb-4bit requires loading Mistral3ForConditionalGeneration as the text encoder and the transformer in bfloat16 precision with CPU offloading enabled. During image generation with a prompt and 50 inference steps, the pipeline encodes text embeddings through the Mistral 3 model before executing diffusion operations. However, executing the pipeline on a GPU with 14.56 GiB capacity triggers a CUDA OutOfMemoryError during 4-bit dequantization inside bitsandbytes matrix multiplication operations. Additionally, the execution logs warn that Flax classes are deprecated in Diffusers and will be removed in version 1.0.0.

Context

Running 4-bit quantized FLUX-2 text-to-image generation using Diffusers and Transformers on GPU hardware with CPU model offloading.

Approach / What changed

Loading Flux2Transformer2DModel and Mistral3ForConditionalGeneration from diffusers/FLUX.2-dev-bnb-4bit with bfloat16 precision, configuring Flux2Pipeline with CPU offload, and generating an image via prompt encoding and diffusion steps.

Takeaways

• FLUX.2 pipeline integration in Diffusers pairs Flux2Transformer2DModel with Mistral3ForConditionalGeneration as the text encoder.
• Inference on a 14.56 GiB GPU with 4-bit bitsandbytes quantization and CPU offloading encountered a CUDA OutOfMemoryError during linear dequantization.
• Flax classes are deprecated across Diffusers and are scheduled to be removed in version 1.0.0.

Related reading

• Two Sigma · Mar 21, 2024

  NeurIPS 2023: Our Favorite Papers on LLMs, Statistical Learning, and More

  Researchers reviewed prominent machine learning papers presented at NeurIPS 2023 covering large language models and statistical learning theory. One investigation showed that claimed emergent abilities in models such as GPT-3 often result from nonlinear evaluation metrics rather than fundamental shifts in model capability. To reduce the computational burden of model adaptation, QLoRA enables 65-billion-parameter model fine-tuning on a single 48-gigabyte GPU via 4-bit NormalFloat quantization and paged optimization. Direct Preference Optimization eliminates complex reward modeling by casting reinforcement learning from human feedback into a preference classification task. Additional work resolved statistical anomalies like double descent using effective parameter counts and introduced stochastic gradient approximations for Gaussian processes.

  Emily Majewski

• huggingface.co · Sep 24, 2021

  Summer at Hugging Face

  Hugging Face released several major updates across its ecosystem, expanding its public Hub repository to over 16,000 models. Platform additions include Spaces for deploying Gradio and Streamlit demo applications, automatic TensorBoard instances, and evaluation metric tracking integrated with Papers With Code leaderboards. The Transformers library gained JAX/Flax support across more than 5,000 models, improved TensorFlow implementations, and introduced the transformers.onnx export module for model conversion. In research, the BigScience project completed large-scale training of a 13-billion-parameter English decoder model on Jean Zay, while the DeDLOC method enabled training the sahajBERT Bengali model without HPC infrastructure. Accepted conference papers detailed the Datasets library, prompt data-point equivalence, prompt-based fine-tuning heuristics, and block pruning techniques that reduced BERT size by 74 percent.