Bringing Nunchaku 4-bit Diffusion Inference to Diffusers

Summary

Modern diffusion transformers running in BF16 often require 20–30 GB of VRAM, limiting access on consumer GPUs; the integration brings Nunchaku Lite checkpoints into Diffusers to reduce memory and denoising latency. SVDQuant handles activation outliers by moving them into weights, preserving a small 16-bit low-rank branch while quantizing residual weights to 4-bit, enabling W4A4 compute in transformer attention and MLP projections. Nunchaku Lite patches standard nn.Linear modules at runtime, loads Hub-hosted CUDA kernels through the kernels package, and exposes ordinary from_pretrained() loading without local compilation; AWQ W4A16 handles modulation and normalization projections. On an RTX 5090, the cited NVFP4 checkpoint generates 1024×1024 images in about 1.7 seconds at roughly 12 GB peak memory versus about 24 GB for BF16, while the bare-bones Nunchaku Lite path delivers around 30% speedup; torch.compile raises reported end-to-end speedup from 1.35x to 1.8x.

Takeaways

1. 1

   SVDQuant uses 4-bit weights and activations with a small 16-bit low-rank correction for outliers; fused kernels combine the low-rank and 4-bit paths to reduce memory-access overhead.

2. 2

   NVFP4 Nunchaku Lite checkpoints support Blackwell GPUs, while INT4 variants support Turing, Ampere, and Ada GPUs; Volta and Hopper are not supported by the 4-bit kernels.

3. 3

   Nunchaku Lite preserves the dense model’s module structure for ordinary Diffusers features, but lacks architecture-specific fused paths and therefore does not match the speed of the original Nunchaku engine.