How Superhuman and Databricks built a 200K QPS inference platform together

Summary

Superhuman replaced a DIY vLLM-based serving stack for its grammar-correction model with Databricks model serving, targeting peak traffic above 200,000 QPS, sub-second P99 latency, four-nines reliability, and no quality regression. The platform combines an Endpoint Discovery Service with power-of-two-choices load balancing, concurrency-based autoscaling, and asymmetric scale policies to reduce hotspots and handle rapid traffic ramps. Lazy-loading container images changed startup from several minutes to a few seconds by fetching metadata first and retrieving 4MB blocks on demand. Runtime work on H100 GPUs raised per-pod throughput from 750 to 1,200 QPS: FP8 weight quantization contributed up to 30%, while a multiprocessing server added 20%, alongside C++ and asynchronous scheduling optimizations. The teams validated changes through shadow testing and evaluation harnesses, and Superhuman is migrating additional models and workflows to the platform.

Takeaways

1. 1

   The Endpoint Discovery Service samples two candidate pods and routes each request to the one with fewer active requests, preventing the hotspots observed with Kubernetes round robin at high QPS.

2. 2

   Lazy-loading uses a seekable block-device image with 4MB sectors; metadata is mounted first, while file blocks are fetched on demand and cached locally.

3. 3

   On H100 GPUs, throughput rose from 750 to 1,200 QPS per pod; FP8 weight quantization delivered up to 30% improvement and multiprocessing CPU dispatch added 20%.