---
title: "Databricks Network Configuration delivery to Tens of Millions of Serverless VMs"
description: "Databricks’ serverless platform launches tens of millions of VMs daily, and each VM needs network configuration before serving customer workloads. The original design synchronously aggregated data from multiple upstream services on the cluster-creation critical path, producing 5,000-ms p99 RPC latency and availability bottlenecks as request volume reached billions per day. The replacement uses change events through a message queue, asynchronous per-workspace recomputation, partition-local snapshots, and periodic reconciliation to recover from missed events. The serving path now performs a single snapshot-store read without upstream calls, while events carry workspace and resource identifiers rather than customer data. After rollout, p99 RPC latency fell to 125 ms, server success rate rose to 99.99%, upstream call volume dropped 86%, and the legacy synchronous framework was deprecated."
---

# Databricks Network Configuration delivery to Tens of Millions of Serverless VMs

[Databricks](https://yomu.fyi/company/databricks) · Manish Bansal, Yankai Zhang, Chen He · Aug 12, 2026

**Type:** Problem & solution

## Summary

Databricks’ serverless platform launches tens of millions of VMs daily, and each VM needs network configuration before serving customer workloads. The original design synchronously aggregated data from multiple upstream services on the cluster-creation critical path, producing 5,000-ms p99 RPC latency and availability bottlenecks as request volume reached billions per day. The replacement uses change events through a message queue, asynchronous per-workspace recomputation, partition-local snapshots, and periodic reconciliation to recover from missed events. The serving path now performs a single snapshot-store read without upstream calls, while events carry workspace and resource identifiers rather than customer data. After rollout, p99 RPC latency fell to 125 ms, server success rate rose to 99.99%, upstream call volume dropped 86%, and the legacy synchronous framework was deprecated.

## Context

Databricks’ serverless compute platform launches tens of millions of VMs daily across AWS, Azure, and GCP. Each VM requires network configuration assembled from multiple upstream services, including storage destinations, private endpoints, Unity Catalog changes, and Delta Sharing destinations. Synchronous aggregation on cluster startup created 5,000 ms p99 latency, compounded upstream availability risk, duplicated computation, and increasing load as usage grew.

## Approach / What changed

The re-architected system consumes upstream change events through a message queue, identifies affected workspaces, and asynchronously recomputes their configurations. Each service partition stores workspace snapshots locally, while a periodic reconciler resynchronizes all workspaces if events are missed. The serving path reads a configuration directly from the snapshot store, avoiding upstream calls; events contain only workspace and resource identifiers.

## Takeaways

- The serving path was reduced from a multi-service dependency chain to a single snapshot-store read, lowering RPC p99 latency from approximately 5,000 ms to 125 ms.
- A periodic reconciler provides eventual consistency and recovers from missed events, while event-driven updates avoid recomputing configurations on every request.
- Partition-local computation and storage reduce cross-partition serving dependencies and limit the blast radius of incidents.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Performance](https://yomu.fyi/topic/performance), [Reliability](https://yomu.fyi/topic/reliability), [Scalability](https://yomu.fyi/topic/scalability)

- Source: [Databricks](https://www.databricks.com/blog/databricks-network-configuration-delivery-tens-millions-serverless-vms)
- Source URL: https://www.databricks.com/blog/databricks-network-configuration-delivery-tens-millions-serverless-vms
- Ingested by Yomu: 2026-08-30T16:51:27.066Z

[Read original post](https://www.databricks.com/blog/databricks-network-configuration-delivery-tens-millions-serverless-vms)