Optimize vLLM speculative decoding with FastMTP heads

Summary

Autoregressive LLM inference is memory-bandwidth bound because each token requires a full pass over billions of parameters, while speculative decoding can verify several drafted tokens in one pass. Speculators 0.6.0 adapts a single native MTP head with FastMTP-style recursive fine-tuning, addressing the mismatch between teacher-forced training and vLLM’s autoregressive reuse of the head. The training loop fuses each next-token embedding with the current hidden state, predicts the following token, feeds the resulting hidden state back, and weights three steps with normalized exponential decay using β = 0.6. Native mtp.* weights extraction, shared full-vocabulary embeddings and output head, and vLLM-compatible stitching support deployment; on Qwen3-Next-80B-A3B-Instruct evaluated on GSM8K, mean accepted length improved at positions 0–2 from 0.897, 0.719, 0.476 to 0.912, 0.776, 0.616.

Takeaways

1. 1

   FastMTP-style fine-tuning recursively feeds an MTP layer’s output hidden state into the next drafting step, matching how vLLM reuses a single head during serving.

2. 2

   The default three-step loss uses normalized exponential decay with β = 0.6, producing weights of approximately 0.51, 0.31, and 0.18.

3. 3

   On Qwen3-Next-80B-A3B-Instruct evaluated on GSM8K, fine-tuning raised mean accepted length at positions 0, 1, and 2 from 0.897, 0.719, and 0.476 to 0.912, 0.776, and 0.616.