---
title: "AI Serving Platform That Adapts to Your Model"
description: "Databricks Custom Model Serving addresses the operational burden of serving custom models, whose resource profiles, traffic patterns, and latency requirements vary widely from small CPU classifiers to large GPU-backed language models. Its fully managed platform packages MLflow models and uses isolated Kubernetes deployments, model-appropriate runtimes, and a short request path to limit interference and per-request overhead. At the center, the AutoPilot Pod Autoscaler combines active concurrency and queue signals for horizontal scaling with CPU, GPU, and memory measurements for model-aware target-concurrency adjustment, allowing one controller to adapt across workloads. Warm pools, provisioned concurrency, and zero-downtime updates address startup and deployment concerns, while reported production results include 90%+ cost savings for some customers, up to 2x improvement in p99 and p50 latency, 100K+ QPS, and 99.99% availability."
---

# AI Serving Platform That Adapts to Your Model

[Databricks](https://yomu.fyi/company/databricks) · Anshul Gupta · Jun 10, 2026

**Type:** Problem & solution

## Summary

Databricks Custom Model Serving addresses the operational burden of serving custom models, whose resource profiles, traffic patterns, and latency requirements vary widely from small CPU classifiers to large GPU-backed language models. Its fully managed platform packages MLflow models and uses isolated Kubernetes deployments, model-appropriate runtimes, and a short request path to limit interference and per-request overhead. At the center, the AutoPilot Pod Autoscaler combines active concurrency and queue signals for horizontal scaling with CPU, GPU, and memory measurements for model-aware target-concurrency adjustment, allowing one controller to adapt across workloads. Warm pools, provisioned concurrency, and zero-downtime updates address startup and deployment concerns, while reported production results include 90%+ cost savings for some customers, up to 2x improvement in p99 and p50 latency, 100K+ QPS, and 99.99% availability.

## Context

Custom models have widely different resource profiles, traffic patterns, and latency budgets, while traffic can spike, burst, or drop to zero. Traditional serving platforms require customers to configure replica counts, concurrency, and autoscaling thresholds, creating ongoing re-profiling work and tension between low latency, high scale, and cost efficiency.

## Approach / What changed

Custom Model Serving uses isolated Kubernetes deployments, model-specific inference runtimes, and a short request path. Its AutoPilot Pod Autoscaler combines request-based horizontal scaling with model-aware vertical scaling that adjusts target concurrency using CPU, GPU, memory, queue, and concurrency signals. Warm pools, provisioned concurrency, and zero-downtime updates address cold starts and releases.

## Takeaways

- The AutoPilot Pod Autoscaler learns how much load each model replica can handle and adjusts target\_concurrency without changing the underlying hardware.
- Horizontal scaling responds to active concurrent requests, while model-aware vertical scaling uses resource and load measurements to account for differences between workloads such as CPU classifiers and GPU-heavy LLMs.
- Reported production results include 90%+ cost savings for some customers, up to 2x improvements in p99 and p50 latency, 100K+ QPS, and 99.99% availability.

**Tags:** [Autoscaling](https://yomu.fyi/topic/autoscaling), [Kubernetes](https://yomu.fyi/topic/kubernetes), [MLflow](https://yomu.fyi/topic/mlflow), [Model Serving](https://yomu.fyi/topic/model-serving)

- Source: [Databricks](https://www.databricks.com/blog/ai-serving-platform-adapts-your-model)
- Source URL: https://www.databricks.com/blog/ai-serving-platform-adapts-your-model
- Ingested by Yomu: 2026-08-31T03:31:03.926Z

[Read original post](https://www.databricks.com/blog/ai-serving-platform-adapts-your-model)