---
title: "Enterprise-Grade Precision for Long-Context Multimodal Embedding Inference on Cloud TPU"
description: "Embedding models convert text, images, and audio into dense vectors used for semantic search, recommendation, intent classification, content discovery, and clustering. The post describes production support for long-context Qwen3 Embedding models on Cloud TPUs through native vLLM TPU integration and GKE Custom Compute Classes, allowing prioritized autoscaling across TPU and GPU capacity pools. Its implementation pads vocabulary tensors for hardware-safe sharding, hardens lazy loading, pre-warms sharding-aware JAX/XLA compilation caches, and uses a hybrid StepPool with CachedRequestState to preserve pooling state during chunked prefill and preemption. Parity evaluations target cosine similarity of at least 0.999 for text and 0.995 for multimodal inputs; on TPU Ironwood, Qwen3-Embedding-8B reached 83,996 total tokens per second and 5.13 requests per second at 16K+ sequence length with bf16 and TP=4."
---

# Enterprise-Grade Precision for Long-Context Multimodal Embedding Inference on Cloud TPU

[Google](https://yomu.fyi/company/google) · Anthony Su, Injae Kwak · Aug 26, 2026

**Type:** Benchmark

## Summary

Embedding models convert text, images, and audio into dense vectors used for semantic search, recommendation, intent classification, content discovery, and clustering. The post describes production support for long-context Qwen3 Embedding models on Cloud TPUs through native vLLM TPU integration and GKE Custom Compute Classes, allowing prioritized autoscaling across TPU and GPU capacity pools. Its implementation pads vocabulary tensors for hardware-safe sharding, hardens lazy loading, pre-warms sharding-aware JAX/XLA compilation caches, and uses a hybrid StepPool with CachedRequestState to preserve pooling state during chunked prefill and preemption. Parity evaluations target cosine similarity of at least 0.999 for text and 0.995 for multimodal inputs; on TPU Ironwood, Qwen3-Embedding-8B reached 83,996 total tokens per second and 5.13 requests per second at 16K+ sequence length with bf16 and TP=4.

## Context

Scaling embedding pipelines to millions of queries creates accelerator-capacity and cost/performance challenges. The workloads also require long contexts, including 4K+ tokens for text and up to 15K+ tokens for multimodal text-and-image inputs, while preserving mathematical parity across heterogeneous hardware.

## Approach / What changed

Native TPU support in vLLM is combined with GKE Custom Compute Classes for prioritized autoscaling across accelerator pools. The implementation adds hardware-safe vocabulary padding, lazy-loading compatibility changes, sharding-aware JAX/XLA compilation pre-warming, and a hybrid StepPool that stores pooling metadata in CachedRequestState across chunked-prefill steps and request preemptions.

## Takeaways

- GKE Custom Compute Classes can apply priority rules to scale across TPU capacity and secondary GPU spot or on-demand pools when the primary TPU reservations are fully utilized.
- The TPU implementation uses vocabulary padding for tensor alignment, promotes attributes during unquantization for lazy loading, and pre-warms sharding-aware JAX/XLA compilation caches.
- The stated parity thresholds are at least 0.999 cosine similarity for text and 0.995 for multimodal inputs; Qwen3-Embedding-8B reached 83,996 total token/s and 5.13 req/s on TPU Ironwood with bf16, TP=4, and 16K+ sequence length.

**Tags:** [Inference](https://yomu.fyi/topic/inference), [TPU](https://yomu.fyi/topic/tpu), [vLLM](https://yomu.fyi/topic/vllm)

- Source: [Google](https://developers.googleblog.com/enterprise-grade-precision-for-long-context-multimodal-embedding-inference-on-cloud-tpu)
- Source URL: https://developers.googleblog.com/enterprise-grade-precision-for-long-context-multimodal-embedding-inference-on-cloud-tpu
- Ingested by Yomu: 2026-09-09T14:27:21.880Z

[Read original post](https://developers.googleblog.com/enterprise-grade-precision-for-long-context-multimodal-embedding-inference-on-cloud-tpu)