---
title: "Announcing New vLLM and SGLang Inference Engines on TPU"
description: "Google announced new PyTorch-native backends for vLLM and SGLang inference engines on TPU, co-developed with Inferact and RadixArk. Built on TorchTPU, the engines make TPU a native PyTorch device with standard tensor management, eager execution, and torch.compile via XLA, using a platform-plugin design that reuses each framework’s worker model and scheduler. They support production workloads including agentic multi-turn, interactive low-latency, high-throughput batch, long context, disaggregated serving, and multimodal models, with custom Pallas kernels called as PyTorch custom operators. Features include automatic prefix caching, speculative decoding with MTP, EAGLE-3, DFlash, DSpark and n-gram drafters, KV cache offloading to host memory and Managed Lustre, and TPU Sync for efficient cache transfer between hosts. Both engines are running in production at Meta and preview customers, with public benchmarks showing industry-leading cost per million tokens at 2.01 cents on Ironwood TPU, and a community contribution TPU capacity pool is available for developers."
---

# Announcing New vLLM and SGLang Inference Engines on TPU

[Google](https://yomu.fyi/company/google) · Qi Zhou, Chris Chan · Oct 20, 2026

**Type:** Announcement

## Summary

Google announced new PyTorch-native backends for vLLM and SGLang inference engines on TPU, co-developed with Inferact and RadixArk. Built on TorchTPU, the engines make TPU a native PyTorch device with standard tensor management, eager execution, and torch.compile via XLA, using a platform-plugin design that reuses each framework’s worker model and scheduler. They support production workloads including agentic multi-turn, interactive low-latency, high-throughput batch, long context, disaggregated serving, and multimodal models, with custom Pallas kernels called as PyTorch custom operators. Features include automatic prefix caching, speculative decoding with MTP, EAGLE-3, DFlash, DSpark and n-gram drafters, KV cache offloading to host memory and Managed Lustre, and TPU Sync for efficient cache transfer between hosts. Both engines are running in production at Meta and preview customers, with public benchmarks showing industry-leading cost per million tokens at 2.01 cents on Ironwood TPU, and a community contribution TPU capacity pool is available for developers.

## Context

Google launched TorchTPU to run PyTorch natively on TPU, then built new vLLM and SGLang inference engines on top of it to serve agentic and multimodal workloads with production-ready features.

## Approach / What changed

The approach is to create PyTorch-native backends for vLLM and SGLang on TorchTPU, using a platform-plugin design that reuses each framework’s worker model and scheduler while adding TPU-specific optimizations in kernels and layer implementations, with custom Pallas kernels called as PyTorch custom operators and TPU Sync for KV cache transfer.

## Takeaways

- The new engines support automatic prefix caching, speculative decoding, KV cache offloading, and disaggregated serving for agentic and multimodal workloads.
- Both engines use TPU Sync, an open-source C++ library, for efficient KV cache transfer between hosts without extra staging copies.
- The engines are already running in production at Meta and preview customers, with public benchmarks showing 2.01 cents per million tokens on Ironwood TPU.

## FAQ

### What is TorchTPU?

TorchTPU makes TPU a native PyTorch device where tensors are standard PyTorch tensors on a tpu device, code runs eagerly or compiles via torch.compile using XLA, and collectives go through torch.distributed.

### How do the engines handle KV cache for disaggregated serving?

They use TPU Sync, an open-source C++ library that reads device buffers behind PyTorch tensors directly, runs transfers on its own threads outside Python while prefill and decode continue, avoiding extra staging copies.

### What models are supported?

Most popular open model families are supported including Qwen, DeepSeek, GLM, Kimi, GPT-OSS, Gemma4 and more, with investment in Day 0 support for new open models.

**Tags:** [Inference](https://yomu.fyi/topic/inference), [PyTorch](https://yomu.fyi/topic/pytorch), [SGLang](https://yomu.fyi/topic/sglang), [TPU](https://yomu.fyi/topic/tpu), [vLLM](https://yomu.fyi/topic/vllm)

- Source: [Google](https://developers.googleblog.com/announcing-new-vllm-and-sglang-inference-engines-on-tpu)
- Source URL: https://developers.googleblog.com/announcing-new-vllm-and-sglang-inference-engines-on-tpu
- Ingested by Yomu: 2026-10-20T18:00:20.837Z

[Read original post](https://developers.googleblog.com/announcing-new-vllm-and-sglang-inference-engines-on-tpu)