---
title: "Optimize vLLM speculative decoding with FastMTP heads"
description: "Autoregressive LLM inference is memory-bandwidth bound because each token requires a full pass over billions of parameters, while speculative decoding can verify several drafted tokens in one pass. Speculators 0.6.0 adapts a single native MTP head with FastMTP-style recursive fine-tuning, addressing the mismatch between teacher-forced training and vLLM’s autoregressive reuse of the head. The training loop fuses each next-token embedding with the current hidden state, predicts the following token, feeds the resulting hidden state back, and weights three steps with normalized exponential decay using β = 0.6. Native mtp.* weights extraction, shared full-vocabulary embeddings and output head, and vLLM-compatible stitching support deployment; on Qwen3-Next-80B-A3B-Instruct evaluated on GSM8K, mean accepted length improved at positions 0–2 from 0.897, 0.719, 0.476 to 0.912, 0.776, 0.616."
---

# Optimize vLLM speculative decoding with FastMTP heads

[Red Hat](https://yomu.fyi/company/red-hat) · Rahul Tuli · Sep 8, 2026

**Type:** Tutorial

## Summary

Autoregressive LLM inference is memory-bandwidth bound because each token requires a full pass over billions of parameters, while speculative decoding can verify several drafted tokens in one pass. Speculators 0.6.0 adapts a single native MTP head with FastMTP-style recursive fine-tuning, addressing the mismatch between teacher-forced training and vLLM’s autoregressive reuse of the head. The training loop fuses each next-token embedding with the current hidden state, predicts the following token, feeds the resulting hidden state back, and weights three steps with normalized exponential decay using β = 0.6. Native mtp.\* weights extraction, shared full-vocabulary embeddings and output head, and vLLM-compatible stitching support deployment; on Qwen3-Next-80B-A3B-Instruct evaluated on GSM8K, mean accepted length improved at positions 0–2 from 0.897, 0.719, 0.476 to 0.912, 0.776, 0.616.

## Context

Native MTP heads are trained with teacher forcing for distinct future positions, but inference engines such as vLLM often reuse a single shipped head recursively to draft multiple tokens. Because the head then consumes its own outputs rather than ground-truth inputs, errors compound and acceptance can decline for later speculative tokens.

## Approach / What changed

Speculators 0.6.0 extracts native mtp.\* weights, fine-tunes one MTP head with a teacher-forced recursive loop that mirrors serving, and applies normalized exponential-decay weighting across speculative steps. The speculator shares the verifier’s embed\_tokens and lm\_head, and a stitcher emits the mtp.\* format expected by vLLM for standard deployment.

## Takeaways

- FastMTP-style fine-tuning recursively feeds an MTP layer’s output hidden state into the next drafting step, matching how vLLM reuses a single head during serving.
- The default three-step loss uses normalized exponential decay with β = 0.6, producing weights of approximately 0.51, 0.31, and 0.18.
- On Qwen3-Next-80B-A3B-Instruct evaluated on GSM8K, fine-tuning raised mean accepted length at positions 0, 1, and 2 from 0.897, 0.719, and 0.476 to 0.912, 0.776, and 0.616.

**Tags:** [Inference](https://yomu.fyi/topic/inference), [Machine Learning](https://yomu.fyi/topic/machine-learning), [vLLM](https://yomu.fyi/topic/vllm)

- Source: [Red Hat](https://developers.redhat.com/articles/2026/09/08/optimize-vllm-speculative-decoding-fastmtp-heads)
- Source URL: https://developers.redhat.com/articles/2026/09/08/optimize-vllm-speculative-decoding-fastmtp-heads
- Ingested by Yomu: 2026-09-08T16:00:56.844Z

[Read original post](https://developers.redhat.com/articles/2026/09/08/optimize-vllm-speculative-decoding-fastmtp-heads)