AI Serving Platform That Adapts to Your Model

Summary

Databricks Custom Model Serving addresses the operational burden of serving custom models, whose resource profiles, traffic patterns, and latency requirements vary widely from small CPU classifiers to large GPU-backed language models. Its fully managed platform packages MLflow models and uses isolated Kubernetes deployments, model-appropriate runtimes, and a short request path to limit interference and per-request overhead. At the center, the AutoPilot Pod Autoscaler combines active concurrency and queue signals for horizontal scaling with CPU, GPU, and memory measurements for model-aware target-concurrency adjustment, allowing one controller to adapt across workloads. Warm pools, provisioned concurrency, and zero-downtime updates address startup and deployment concerns, while reported production results include 90%+ cost savings for some customers, up to 2x improvement in p99 and p50 latency, 100K+ QPS, and 99.99% availability.

Takeaways

1. 1

   The AutoPilot Pod Autoscaler learns how much load each model replica can handle and adjusts target_concurrency without changing the underlying hardware.

2. 2

   Horizontal scaling responds to active concurrent requests, while model-aware vertical scaling uses resource and load measurements to account for differences between workloads such as CPU classifiers and GPU-heavy LLMs.

3. 3

   Reported production results include 90%+ cost savings for some customers, up to 2x improvements in p99 and p50 latency, 100K+ QPS, and 99.99% availability.