Boost Inference Performance up to 15x on NVIDIA Blackwell Using DFlash Speculative Decoding

Summary

Sequential token generation in autoregressive LLMs limits GPU utilization and serving throughput, while conventional speculative decoding remains constrained by autoregressive draft models. DFlash addresses this bottleneck by employing a lightweight block-diffusion drafter that predicts an entire block of masked candidate tokens in a single forward pass. The architecture incorporates target hidden-state conditioning and key-value injection into draft model projections while relying on the target model for parallel verification. Evaluated on NVIDIA Blackwell GPUs across TensorRT-LLM, vLLM, and SGLang, DFlash increases gpt-oss-120b throughput by more than 15x at high interactivity targets and delivers up to 5.8x speedups on Gemma 4 31B compared to autoregressive decoding.

Context

Autoregressive LLMs generate tokens sequentially, which limits GPU utilization and constrains throughput in latency-sensitive serving scenarios. While speculative decoding mitigates this by using a lightweight model to draft tokens for parallel verification by a target model, traditional speculative methods rely on autoregressive draft models that still draft sequentially. Consequently, drafting costs grow with the speculative token count, creating a bottleneck on modern high-compute GPU architectures.

Approach / What changed

DFlash implements block-diffusion speculative decoding by replacing sequential draft models with a lightweight drafter that predicts a block of masked future tokens in a single forward pass. The drafter conditions on target model hidden states and injects target context features into its key-value projections across layers to preserve high acceptance rates. The target model then verifies candidate blocks in parallel, maintaining output distribution fidelity across serving frameworks like TensorRT-LLM, vLLM, and SGLang.

Takeaways

• DFlash replaces sequential drafting with single-pass block-diffusion drafting, combining target hidden-state conditioning and multi-layer KV injection while preserving target model output distributions.
• On an eight NVIDIA DGX B300 system running TensorRT-LLM, DFlash delivers over 15x higher throughput for gpt-oss-120b than autoregressive decoding at 500 to 600 tokens/sec per user, beating EAGLE-3 by 1.5x.
• DFlash integrates into vLLM via the Speculators library and into SGLang through configuration updates, achieving up to 5.8x speedup on Gemma 4 31B and 5.1x on Qwen3 8-B on single Blackwell GPUs.

Related reading