From prototype to production: High QPS for Databricks AI Search

Summary

Databricks AI Search now offers generally available high-QPS scaling for Standard endpoints, addressing production workloads such as search bars, recommendations, and real-time entity resolution. Operators set a human-readable target_qps value when creating or updating an endpoint through the Python SDK, REST API, or UI, while Databricks provisions the required compute without manual replica counts, node sizing, or load balancers. Existing Unity Catalog governance and Delta Sync remain in place, and scaling progress is exposed through scaling_info as it moves from SCALING_CHANGE_IN_PROGRESS to SCALING_CHANGE_APPLIED; endpoint observability shows requests per second, latency, and health. Service principal authentication is recommended for high-QPS traffic, while personal access tokens are capped at a few tens of QPS, and new capacity applies when an index is created or synced.

Context

Production search workloads can generate thousands of queries per second, with traffic spikes, latency-sensitive requests, and operational issues such as 429 errors, rising P95 latency, duplicate endpoints, or client-side load balancing.

Approach / What changed

Databricks adds target_qps configuration for Standard endpoints, provisions compute capacity to meet the declared target, exposes scaling progress and endpoint observability, and supports configuration through the Python SDK, REST API, and UI.

Takeaways

• target_qps can be set when creating an endpoint or updated on an existing Standard endpoint; new capacity takes effect when an index is created or synced.
• The scaling_info field reports transitions from SCALING_CHANGE_IN_PROGRESS to SCALING_CHANGE_APPLIED, while the endpoint UI surfaces requests per second, latency, and endpoint health.
• Service principal traffic uses performance-optimized networks for high-QPS workloads; personal access token traffic is capped at a few tens of QPS and is intended for prototyping.

Related reading