Co-Designing AI Models Using Speculative Decoding for Faster LLM Inference

Summary

Speculative decoding accelerates large language model inference by having a smaller draft model propose multiple tokens and a target model verify them in parallel, preserving the standard output sequence under its acceptance criteria. It reduces decoding iterations and increases target-model arithmetic intensity, but speedup depends on acceptance length, verification time, draft latency, batch size, and hardware behavior. For attention-heavy workloads, the recommended starting point is D = 128/G - 1, where G is query heads per KV head; larger D values should align G × (1 + D) with 128-token kernel tiles. The remaining guidance covers compute-bound GEMMs, low-latency draft-cost tradeoffs, and choosing among external models, EAGLE-3, MTP, DFlash, DSpark, and suffix or n-gram methods; SPEED-Bench, TensorRT LLM, and NVIDIA/Model-Optimizer support measurement, training, and quantization workflows.

Takeaways

1. 1

   Speculative decoding can move target-model GEMMs toward the compute-bound region without increasing KV cache capacity pressure.

2. 2

   When attention dominates, D = 128/G - 1 is recommended as a starting point; larger draft lengths should align G × (1 + D) with multiples of 128.

3. 3

   SPEED-Bench measures acceptance length on realistic workloads, while TensorRT LLM can quantify draft overhead and NVIDIA/Model-Optimizer provides examples for EAGLE-3, DFlash, and DSpark.