---
title: "Introducing Modal Auto Endpoints: Optimized inference you actually own"
description: "Modal Auto Endpoints is a self-serve service for production-grade LLM inference that gives teams ownership of the entire serving stack, not just an API. Unlike proprietary providers, it exposes the inference code, engine flags, and GPU selection, and provides detailed metrics like speculative decoding acceptance length and per-replica token latency quantiles in a dashboard. Built on Modal’s infrastructure, it uses autoscaling to handle traffic spikes by spinning up additional replicas automatically, global GPU capacity without capacity management, and Modal Servers for ultra-low-latency routing with only 5ms overhead. For each supported model, it offers a pre-configured deployment using speculative decoding with the DFlash block-diffusion drafter from Z Lab, and teams can benchmark latency and throughput tradeoffs with a click after deployment. The platform is designed with a declarative interface based on workloads and SLOs, with future automation levels including autoinference, autospec, autodistill, and autoresearch for continuous improvement of inference performance."
---

# Introducing Modal Auto Endpoints: Optimized inference you actually own

[Modal](https://yomu.fyi/company/modal) · Jun 23, 2026

**Type:** Announcement

## Summary

Modal Auto Endpoints is a self-serve service for production-grade LLM inference that gives teams ownership of the entire serving stack, not just an API. Unlike proprietary providers, it exposes the inference code, engine flags, and GPU selection, and provides detailed metrics like speculative decoding acceptance length and per-replica token latency quantiles in a dashboard. Built on Modal’s infrastructure, it uses autoscaling to handle traffic spikes by spinning up additional replicas automatically, global GPU capacity without capacity management, and Modal Servers for ultra-low-latency routing with only 5ms overhead. For each supported model, it offers a pre-configured deployment using speculative decoding with the DFlash block-diffusion drafter from Z Lab, and teams can benchmark latency and throughput tradeoffs with a click after deployment. The platform is designed with a declarative interface based on workloads and SLOs, with future automation levels including autoinference, autospec, autodistill, and autoresearch for continuous improvement of inference performance.

## Context

Teams using proprietary LLM APIs face risks of model degradation or access retraction, while rolling custom inference services requires managing engine tuning, benchmarking, container deployment, autoscaling, and metrics.

## Approach / What changed

Modal Auto Endpoints provides an OpenAI API-compatible, production-ready inference service backed by a visible and controllable Modal App. It exposes the inference code, engine flags, and GPU selection, provides detailed engine-level metrics, and offers pre-configured deployments using speculative decoding with DFlash block-diffusion drafters. It runs on Modal’s infrastructure with autoscaling, global GPU capacity, and Modal Servers for ultra-low-latency routing.

## Takeaways

- Modal Auto Endpoints exposes the inference code and engine flags, not just an API, giving teams full ownership and inspectability of the serving stack.
- It provides engine-level observability metrics including TTFT, ITL, queueing, and speculative decoding acceptance length, available in-dashboard and via OTEL export.
- The platform uses speculative decoding with DFlash block-diffusion drafters from Z Lab, achieving over 4x faster than baseline and over 1.5x faster than other speculators on multiple benchmarks.

## FAQ

### How does Modal Auto Endpoints differ from traditional inference providers?

Unlike traditional providers that hide the serving stack, Modal Auto Endpoints exposes the inference code, engine flags, and GPU selection. It provides detailed engine-level metrics and allows deployment via CLI without sales calls, giving teams full ownership and inspectability.

### What metrics are available for debugging inference performance?

Modal provides server metrics like GPU temperature, power, and utilization, and inference metrics like time-to-first-token (TTFT), inter-token latency (ITL), queueing, and speculative decoding acceptance length. These are available in-dashboard and via OTEL export.

### How does speculative decoding work in Auto Endpoints?

Auto Endpoints uses the DFlash block-diffusion drafter architecture from Z Lab with every compatible model. Modal trained and released its own DFlash drafter models to expand support and ensure optimal performance, achieving over 4x faster than baseline on multiple benchmarks.

**Tags:** [Autoscaling](https://yomu.fyi/topic/autoscaling), [Inference](https://yomu.fyi/topic/inference), [LLMs](https://yomu.fyi/topic/llm), [Observability](https://yomu.fyi/topic/observability), [Speculative Decoding](https://yomu.fyi/topic/speculative-decoding)

- Source: [Modal](https://modal.com/blog/introducing-auto-endpoints)
- Source URL: https://modal.com/blog/introducing-auto-endpoints
- Ingested by Yomu: 2026-09-20T07:47:01.852Z

[Read original post](https://modal.com/blog/introducing-auto-endpoints)