How Superhuman and Databricks built a 200K QPS inference platform together

Summary

Superhuman replaced a DIY vLLM-based serving stack for its grammar-correction model with Databricks model serving, targeting peak traffic above 200,000 QPS, sub-second P99 latency, four-nines reliability, and no quality regression. The platform combines an Endpoint Discovery Service with power-of-two-choices load balancing, concurrency-based autoscaling, and asymmetric scale policies to reduce hotspots and handle rapid traffic ramps. Lazy-loading container images changed startup from several minutes to a few seconds by fetching metadata first and retrieving 4MB blocks on demand. Runtime work on H100 GPUs raised per-pod throughput from 750 to 1,200 QPS: FP8 weight quantization contributed up to 30%, while a multiprocessing server added 20%, alongside C++ and asynchronous scheduling optimizations. The teams validated changes through shadow testing and evaluation harnesses, and Superhuman is migrating additional models and workflows to the platform.

Context

Superhuman’s custom grammar-correction model served more than 200,000 QPS at peak, with roughly 50 input and 50 output tokens per request. Its vLLM-based stack required months of manual tuning for each model iteration, while capacity planning, performance tuning, and autoscaling created an increasing operational burden. The team needed sub-second P99 latency, strict four-nines reliability, and no quality regression.

Approach / What changed

Databricks and Superhuman used Databricks model serving with an Endpoint Discovery Service, power-of-two-choices load balancing, dynamic concurrency-based autoscaling, and asymmetric scale policies. They added lazy-loading container images to accelerate startup, then improved per-pod performance through FP8 weight quantization, hybrid-precision support, multiprocessing runtime servers, reduced Python overhead, and asynchronous CPU-GPU scheduling.

Takeaways

• The Endpoint Discovery Service samples two candidate pods and routes each request to the one with fewer active requests, preventing the hotspots observed with Kubernetes round robin at high QPS.
• Lazy-loading uses a seekable block-device image with 4MB sectors; metadata is mounted first, while file blocks are fetched on demand and cached locally.
• On H100 GPUs, throughput rose from 750 to 1,200 QPS per pod; FP8 weight quantization delivered up to 30% improvement and multiprocessing CPU dispatch added 20%.

Related reading