---
title: "How Superhuman and Databricks built a 200K QPS inference platform together"
description: "Superhuman replaced a DIY vLLM-based serving stack for its grammar-correction model with Databricks model serving, targeting peak traffic above 200,000 QPS, sub-second P99 latency, four-nines reliability, and no quality regression. The platform combines an Endpoint Discovery Service with power-of-two-choices load balancing, concurrency-based autoscaling, and asymmetric scale policies to reduce hotspots and handle rapid traffic ramps. Lazy-loading container images changed startup from several minutes to a few seconds by fetching metadata first and retrieving 4MB blocks on demand. Runtime work on H100 GPUs raised per-pod throughput from 750 to 1,200 QPS: FP8 weight quantization contributed up to 30%, while a multiprocessing server added 20%, alongside C++ and asynchronous scheduling optimizations. The teams validated changes through shadow testing and evaluation harnesses, and Superhuman is migrating additional models and workflows to the platform."
---

# How Superhuman and Databricks built a 200K QPS inference platform together

[Databricks](https://yomu.fyi/company/databricks) · Myke Troianovskyi, Christoph Stüber, Wai Wu, Arjun DCunha, Amine El Helou, Tian Ouyang, Jarek Odzga, Alex Coleman · May 8, 2026

**Type:** Problem & solution

## Summary

Superhuman replaced a DIY vLLM-based serving stack for its grammar-correction model with Databricks model serving, targeting peak traffic above 200,000 QPS, sub-second P99 latency, four-nines reliability, and no quality regression. The platform combines an Endpoint Discovery Service with power-of-two-choices load balancing, concurrency-based autoscaling, and asymmetric scale policies to reduce hotspots and handle rapid traffic ramps. Lazy-loading container images changed startup from several minutes to a few seconds by fetching metadata first and retrieving 4MB blocks on demand. Runtime work on H100 GPUs raised per-pod throughput from 750 to 1,200 QPS: FP8 weight quantization contributed up to 30%, while a multiprocessing server added 20%, alongside C++ and asynchronous scheduling optimizations. The teams validated changes through shadow testing and evaluation harnesses, and Superhuman is migrating additional models and workflows to the platform.

## Context

Superhuman’s custom grammar-correction model served more than 200,000 QPS at peak, with roughly 50 input and 50 output tokens per request. Its vLLM-based stack required months of manual tuning for each model iteration, while capacity planning, performance tuning, and autoscaling created an increasing operational burden. The team needed sub-second P99 latency, strict four-nines reliability, and no quality regression.

## Approach / What changed

Databricks and Superhuman used Databricks model serving with an Endpoint Discovery Service, power-of-two-choices load balancing, dynamic concurrency-based autoscaling, and asymmetric scale policies. They added lazy-loading container images to accelerate startup, then improved per-pod performance through FP8 weight quantization, hybrid-precision support, multiprocessing runtime servers, reduced Python overhead, and asynchronous CPU-GPU scheduling.

## Takeaways

- The Endpoint Discovery Service samples two candidate pods and routes each request to the one with fewer active requests, preventing the hotspots observed with Kubernetes round robin at high QPS.
- Lazy-loading uses a seekable block-device image with 4MB sectors; metadata is mounted first, while file blocks are fetched on demand and cached locally.
- On H100 GPUs, throughput rose from 750 to 1,200 QPS per pod; FP8 weight quantization delivered up to 30% improvement and multiprocessing CPU dispatch added 20%.

**Tags:** [Databricks](https://yomu.fyi/topic/databricks), [GPU](https://yomu.fyi/topic/gpu), [Kubernetes](https://yomu.fyi/topic/kubernetes), [Model Serving](https://yomu.fyi/topic/model-serving)

- Source: [Databricks](https://www.databricks.com/blog/how-superhuman-and-databricks-built-200k-qps-inference-platform-together)
- Source URL: https://www.databricks.com/blog/how-superhuman-and-databricks-built-200k-qps-inference-platform-together
- Ingested by Yomu: 2026-08-31T03:35:12.018Z

[Read original post](https://www.databricks.com/blog/how-superhuman-and-databricks-built-200k-qps-inference-platform-together)