---
title: "Co-Designing AI Models Using Speculative Decoding for Faster LLM Inference"
description: "Speculative decoding accelerates large language model inference by having a smaller draft model propose multiple tokens and a target model verify them in parallel, preserving the standard output sequence under its acceptance criteria. It reduces decoding iterations and increases target-model arithmetic intensity, but speedup depends on acceptance length, verification time, draft latency, batch size, and hardware behavior. For attention-heavy workloads, the recommended starting point is D = 128/G - 1, where G is query heads per KV head; larger D values should align G × (1 + D) with 128-token kernel tiles. The remaining guidance covers compute-bound GEMMs, low-latency draft-cost tradeoffs, and choosing among external models, EAGLE-3, MTP, DFlash, DSpark, and suffix or n-gram methods; SPEED-Bench, TensorRT LLM, and NVIDIA/Model-Optimizer support measurement, training, and quantization workflows."
---

# Co-Designing AI Models Using Speculative Decoding for Faster LLM Inference

[NVIDIA](https://yomu.fyi/company/nvidia-developer-blog) · Sep 2, 2026

**Type:** Explainer

## Summary

Speculative decoding accelerates large language model inference by having a smaller draft model propose multiple tokens and a target model verify them in parallel, preserving the standard output sequence under its acceptance criteria. It reduces decoding iterations and increases target-model arithmetic intensity, but speedup depends on acceptance length, verification time, draft latency, batch size, and hardware behavior. For attention-heavy workloads, the recommended starting point is D = 128/G - 1, where G is query heads per KV head; larger D values should align G × (1 + D) with 128-token kernel tiles. The remaining guidance covers compute-bound GEMMs, low-latency draft-cost tradeoffs, and choosing among external models, EAGLE-3, MTP, DFlash, DSpark, and suffix or n-gram methods; SPEED-Bench, TensorRT LLM, and NVIDIA/Model-Optimizer support measurement, training, and quantization workflows.

## Context

Autoregressive LLM decoding generates tokens sequentially, and the optimal speculative-decoding configuration varies with batch size, attention behavior, latency targets, workload, hardware, and draft-model costs.

## Approach / What changed

Use a smaller draft model or another draft mechanism to propose D tokens, verify them in parallel with the target model, and select draft length and mechanism using acceptance length, verification performance, draft overhead, and training and deployment costs.

## Takeaways

- Speculative decoding can move target-model GEMMs toward the compute-bound region without increasing KV cache capacity pressure.
- When attention dominates, D = 128/G - 1 is recommended as a starting point; larger draft lengths should align G × (1 + D) with multiples of 128.
- SPEED-Bench measures acceptance length on realistic workloads, while TensorRT LLM can quantify draft overhead and NVIDIA/Model-Optimizer provides examples for EAGLE-3, DFlash, and DSpark.

**Tags:** [GPU](https://yomu.fyi/topic/gpu), [Inference](https://yomu.fyi/topic/inference), [LLMs](https://yomu.fyi/topic/llm), [Speculative Decoding](https://yomu.fyi/topic/speculative-decoding)

- Source: [NVIDIA](https://developer.nvidia.com/blog/co-designing-ai-models-using-speculative-decoding-for-faster-llm-inference)
- Source URL: https://developer.nvidia.com/blog/co-designing-ai-models-using-speculative-decoding-for-faster-llm-inference
- Ingested by Yomu: 2026-09-02T20:00:53.948Z

[Read original post](https://developer.nvidia.com/blog/co-designing-ai-models-using-speculative-decoding-for-faster-llm-inference)