Loading…
Topic
12 posts about Speculative Decoding. Every summary links to the original.
See the idea
Blackwell inference is a pipeline, not a chip upgrade. This explainer walks five stages — fleet validation, batching, KV caching, low-precision kernels, and speculative decoding — that turn peak FLOPs into cheap tokens.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Hybrid models interleave state-space model (SSM) layers with attention layers, but SSM layers can become a decoding bottleneck at serving batch sizes due to memory-bound state updates. ReplaySSM represents the SSM state as a checkpoint plus a buffer of recent inputs, separating output computation from state updates. This reduces state memory traffic by roughly half, simplifies rollback for speculative decoding by avoiding state snapshots, and supports low-precision state reads without propagating rounding error through the recurrence, enabling separate precision for output computation and state updates. Evaluated on Nemotron 3 and Qwen3.5 models from 4B to 120B parameters on H100 and B300 GPUs, ReplaySSM achieves 1.20× to 1.47× faster decode steps at batch size 256, with end-to-end gains growing with batch size. The technique is integrated into TensorRT-LLM, vLLM, SGLang, and FlashInfer, and runs in production in GLM-5.3-Flash and Kimi-K3.
Liquid AI released an experimental DSpark draft model for their vision-language model LFM2.5-VL-3B. The drafter adds a speculative decoding path that trades a minimal increase in memory footprint for a larger speedup without changing output quality. The 280M-parameter drafter, a 4-layer attention-only transformer, captures the target model’s hidden states at fixed tapped layers to draft a block of candidate tokens. On an M5 Max, decoding runs 2.30x to 3.13x faster by task, with end-to-end latency improvements of 1.56x to 2.62x. The drafter ships with day-one support for llama.cpp, MLX-VLM, and SGLang, and is available on Hugging Face in Safetensors and GGUF formats.
Junie Local’s update introduces Qwen3.8-3.6-27B-blend, a merged model that balances speed and capability for local coding agents. The blend, created by merging Qwen3.6 and Qwen3.8 weights in equal proportions, completed 37 tasks in a 100-task internal benchmark, compared to 34 for Qwen3.6 and 39 for Qwen3.8, while generating 71% fewer output tokens than Qwen3.8. The runtime uses multi-token prediction (MTP), with a two-token proposal on M5 MacBook Pro making decoding 60% faster. A four-bit MTP head was kept over an eight-bit one due to similar acceptance rates and memory savings. The update also brings experimental NVIDIA RTX support on Windows for machines with at least 24 GB of VRAM.
Production serving for agentic AI must maximize concurrent users on fixed GPU infrastructure while preserving latency, especially with long prompts, reused context, and streamed responses. The article presents NVIDIA NIM as a deployable microservice that packages validated model- and GPU-aware choices, then measures Nemotron 3 Ultra NIM 2.0.12 against an open-source baseline on four B200 GPUs using a 64K/400 workload, 76% KV reuse, and 50 TPS per user (20 ms ITL). The optimized stack combines autotuned MoE and Mamba kernels, tensor and expert-aware parallelism, prefix and Mamba state-cache reuse, scheduling, batching, memory tuning, and MTP speculative decoding. At the stated target, throughput rises from 718 to 1,997 output tokens per second—more than 2.5x—while the article recommends replaying representative traffic with NVIDIA AIPerf to select a Pareto point meeting the application’s latency constraint.
Reasoning and agentic models have typically required data-center inference, creating network dependencies, higher costs, and potential data exposure for edge applications. The post explains how to choose between dense Qwen3.8-27B, which activates all 27 billion parameters, and mixture-of-experts Nemotron 3.5 Lightning, which has 30 billion total parameters but activates 3 billion per token. Deployment uses vLLM on Jetson with NVFP4 quantization and speculative decoding, where a draft model proposes tokens and the main model verifies them; the examples include DSpark for Nemotron and DFlash2 for Qwen. Reported tests show up to 6.28x decode-throughput speedup over BF16, while workload-specific throughput still varies. Validation with representative prompts is recommended before deployment, and custom quantization or speculator training is reserved for cases where accuracy or acceptance rates reveal a clear gap.
Speculative decoding accelerates large language model inference by having a smaller draft model propose multiple tokens and a target model verify them in parallel, preserving the standard output sequence under its acceptance criteria. It reduces decoding iterations and increases target-model arithmetic intensity, but speedup depends on acceptance length, verification time, draft latency, batch size, and hardware behavior. For attention-heavy workloads, the recommended starting point is D = 128/G - 1, where G is query heads per KV head; larger D values should align G × (1 + D) with 128-token kernel tiles. The remaining guidance covers compute-bound GEMMs, low-latency draft-cost tradeoffs, and choosing among external models, EAGLE-3, MTP, DFlash, DSpark, and suffix or n-gram methods; SPEED-Bench, TensorRT LLM, and NVIDIA/Model-Optimizer support measurement, training, and quantization workflows.
Moonshot released Kimi K3, a 2.8 trillion parameter multimodal mixture-of-experts transformer with a 1 million token context window and native vision, now on Modal. Modal runs the model at 460 tokens per second on release day through a partnership with Moonshot and vLLM, with token-based pricing on a Shared API or dedicated Auto Endpoint. K3 activates 16 of 896 experts per token and uses Kimi Delta Attention along with Attention Residuals to achieve roughly 2.5 times the scaling efficiency of its predecessor K2. Moonshot employed quantization-aware training from the SFT stage with MXFP4 weights and MXFP8 activations, and contributed a new prefix caching implementation to vLLM to address challenges from KDA. Modal deployed a custom DFlash speculative decoding drafter tuned to K3’s architecture, yielding 360 percent faster interactivity and 88 percent higher throughput on agentic workloads.
Thinking Machines released Inkling, a general-purpose multimodal model that accepts text, image and audio inputs and generates text outputs. The 975B-parameter mixture-of-experts transformer activates 41B parameters per token, uses a 1M-token context window, and employs local attention: five sliding window layers for every global layer. Modal provided day-zero support with a custom DFlash block-diffusion speculator, achieving 250 tokens per second per user on 8x B200s with 67% faster interactivity than the model’s built-in multi-token prediction path. The DFlash drafter uses all local attention and causal layers, keeping cost flat as block size grows. Inkling is available today as an OpenAI-compatible shared endpoint through Modal Auto Endpoints with token-based pricing, allowing users to try the model interactively.
Multi-Token Residual Prediction (MRP) adapts multi-token prediction for diffusion language models by training a small module to predict the residual between adjacent denoising steps rather than the full next-step distribution. This residual target is easier to learn, allowing a tiny 3-layer transformer to accurately predict corrections across multiple steps. In the static regime, MRP enables lossless speculative decoding with up to 1.56× throughput in SGLang, or direct decoding with a small quality cost. In the dynamic regime, MRP recovers up to +16 accuracy points lost by aggressive low-threshold decoding through a remasking mechanism. The module is trained on a frozen backbone and composes cleanly with existing inference methods.
Modal launched Auto Endpoints, combining speculative decoding with Blackwell GPUs, SGLang engine, and Modal Servers to achieve low inference latency for production deployments. The decode phase, where each token requires a full forward pass loading model weights, is the primary latency source, making speculative decoding the most impactful optimization. Speculative decoding runs a separate DFlash speculator model in parallel with the target model using KV projections and a custom Triton kernel, achieving near-linear speedup proportional to acceptance length. In the Decagon voice case study, Modal cut 100ms from a 290ms baseline by mid-training a custom speculator on synthetic data, beating proprietary providers by 60ms. Modal plans to automate speculator training with continuous improvement on production data, moving toward autospec for Auto Endpoints.
Modal Auto Endpoints is a self-serve service for production-grade LLM inference that gives teams ownership of the entire serving stack, not just an API. Unlike proprietary providers, it exposes the inference code, engine flags, and GPU selection, and provides detailed metrics like speculative decoding acceptance length and per-replica token latency quantiles in a dashboard. Built on Modal’s infrastructure, it uses autoscaling to handle traffic spikes by spinning up additional replicas automatically, global GPU capacity without capacity management, and Modal Servers for ultra-low-latency routing with only 5ms overhead. For each supported model, it offers a pre-configured deployment using speculative decoding with the DFlash block-diffusion drafter from Z Lab, and teams can benchmark latency and throughput tradeoffs with a click after deployment. The platform is designed with a declarative interface based on workloads and SLOs, with future automation levels including autoinference, autospec, autodistill, and autoresearch for continuous improvement of inference performance.
Speculative decoding is presented as the most important optimization for LLM inference, delivering speedups of 2x-3x rather than small percentage gains from kernel tuning. The technique accelerates the decode phase by using a lightweight draft model to generate candidate tokens that the target model processes in parallel, with lossless resampling. The authors released DFlash speculators for multiple Qwen models, achieving an additional 5-20% speedup over existing baselines, and worked with SGLang to integrate them. On a B200 node, the Qwen 3.5 122B-A10B model with speculation reaches over 1000 tok/s at concurrency 1, compared to 250 tok/s without speculation, demonstrating a 4x speedup. Training these speculators is simplified because the data-generating process is the target model itself, making data collection trivial, and the post envisions a flywheel where custom speculators and distillation enable continuous improvement in latency, cost, and quality for self-hosted inference.
The best engineering reads, once a week.
