---
title: "Achieve state-of-the-art inference latencies with speculative decoding"
description: "Modal launched Auto Endpoints, combining speculative decoding with Blackwell GPUs, SGLang engine, and Modal Servers to achieve low inference latency for production deployments. The decode phase, where each token requires a full forward pass loading model weights, is the primary latency source, making speculative decoding the most impactful optimization. Speculative decoding runs a separate DFlash speculator model in parallel with the target model using KV projections and a custom Triton kernel, achieving near-linear speedup proportional to acceptance length. In the Decagon voice case study, Modal cut 100ms from a 290ms baseline by mid-training a custom speculator on synthetic data, beating proprietary providers by 60ms. Modal plans to automate speculator training with continuous improvement on production data, moving toward autospec for Auto Endpoints."
---

# Achieve state-of-the-art inference latencies with speculative decoding

[Modal](https://yomu.fyi/company/modal) · Jun 24, 2026

**Type:** Problem & solution

## Summary

Modal launched Auto Endpoints, combining speculative decoding with Blackwell GPUs, SGLang engine, and Modal Servers to achieve low inference latency for production deployments. The decode phase, where each token requires a full forward pass loading model weights, is the primary latency source, making speculative decoding the most impactful optimization. Speculative decoding runs a separate DFlash speculator model in parallel with the target model using KV projections and a custom Triton kernel, achieving near-linear speedup proportional to acceptance length. In the Decagon voice case study, Modal cut 100ms from a 290ms baseline by mid-training a custom speculator on synthetic data, beating proprietary providers by 60ms. Modal plans to automate speculator training with continuous improvement on production data, moving toward autospec for Auto Endpoints.

## Context

Inference latency is critical for conversational AI agents, where every millisecond saved adds headroom for increased intelligence or reduced costs. The decode phase of transformer inference, where each output token requires a full forward pass loading model weights, is the primary latency source. Speculative decoding parallelizes token generation and is the most impactful optimization, yielding speedups roughly linear in acceptance length. Modal’s Auto Endpoints are designed for this use case.

## Approach / What changed

Modal combines Blackwell GPUs, the SGLang inference engine, and Modal Servers with regional proxies for low communication latency. They reduce host overhead by profiling and optimizing engine synchronization. The primary latency win comes from speculative decoding using a custom DFlash speculator model that reuses KV projections from the target model. This speculator is mid-trained on synthetic data generated by the target model to avoid exposing sensitive user data. A custom Triton kernel enables the DFlash technique. The acceptance length of the speculator determines speedup, which is near-linear.

## Takeaways

- Speculative decoding with a custom DFlash speculator model that uses KV projections from the target model can achieve near-linear speedup proportional to acceptance length, making it the most impactful optimization for low latency inference.
- Mid-training a speculator model on synthetic data generated by the target model avoids exposing sensitive production data while effectively teaching the speculator to predict the target’s outputs, especially for structured tasks like tool-calling.
- In the Decagon case study, combining communication latency reduction through Modal Servers, host overhead reduction via engine optimization, and a custom speculator cut 100ms off end-to-end latency, beating proprietary providers by 60ms.

## FAQ

### Why is speculative decoding the most important optimization for low latency inference?

Because the decode phase, which involves sequential token generation, is the primary source of latency in transformer inference. Speculative decoding parallelizes token generation by using a separate draft model, achieving near-linear speedup proportional to acceptance length, while kernel and host optimizations yield only incremental improvements.

### How do you train a custom speculator model without exposing user data?

You use synthetic data generated by passing public prompts through the target model to obtain structured completions that mimic production patterns. This avoids exposing sensitive data while still teaching the speculator to predict the target's outputs, especially for tasks with strong grammars like tool-calling or code generation.

### How much latency improvement did Modal achieve in the Decagon case study?

Modal cut 100ms from a baseline of 290ms (p50) by applying a custom DFlash speculator model mid-trained on synthetic data, beating the fastest proprietary inference provider by 60ms.

**Tags:** [GPU](https://yomu.fyi/topic/gpu), [Inference](https://yomu.fyi/topic/inference), [LLMs](https://yomu.fyi/topic/llm), [SGLang](https://yomu.fyi/topic/sglang), [Speculative Decoding](https://yomu.fyi/topic/speculative-decoding)

- Source: [Modal](https://modal.com/blog/achieve-sota-specdec)
- Source URL: https://modal.com/blog/achieve-sota-specdec
- Ingested by Yomu: 2026-09-20T07:47:03.644Z

[Read original post](https://modal.com/blog/achieve-sota-specdec)