Announcing New vLLM and SGLang Inference Engines on TPU

Summary

Google announced new PyTorch-native backends for vLLM and SGLang inference engines on TPU, co-developed with Inferact and RadixArk. Built on TorchTPU, the engines make TPU a native PyTorch device with standard tensor management, eager execution, and torch.compile via XLA, using a platform-plugin design that reuses each framework’s worker model and scheduler. They support production workloads including agentic multi-turn, interactive low-latency, high-throughput batch, long context, disaggregated serving, and multimodal models, with custom Pallas kernels called as PyTorch custom operators. Features include automatic prefix caching, speculative decoding with MTP, EAGLE-3, DFlash, DSpark and n-gram drafters, KV cache offloading to host memory and Managed Lustre, and TPU Sync for efficient cache transfer between hosts. Both engines are running in production at Meta and preview customers, with public benchmarks showing industry-leading cost per million tokens at 2.01 cents on Ironwood TPU, and a community contribution TPU capacity pool is available for developers.

Takeaways

1. 1

   The new engines support automatic prefix caching, speculative decoding, KV cache offloading, and disaggregated serving for agentic and multimodal workloads.

2. 2

   Both engines use TPU Sync, an open-source C++ library, for efficient KV cache transfer between hosts without extra staging copies.

3. 3

   The engines are already running in production at Meta and preview customers, with public benchmarks showing 2.01 cents per million tokens on Ironwood TPU.

FAQ