Rethinking Distributed Systems for Serverless Performance and Reliability

Summary

Building serverless compute for Apache Spark requires more than warm machine pools or basic autoscaling because traditional deployments couple user applications to infrastructure, allowing contention and inefficiencies to undermine performance and reliability. The proposed architecture separates these concerns through Spark Connect’s client-server model over gRPC, a gateway that routes workloads using query size, cluster utilization, and latency profile, and an adaptive autoscaler that adjusts capacity horizontally and vertically. Spark Connect isolates user applications from drivers, while the gateway continually re-evaluates placement to reduce interference between workloads. The autoscaler offers Standard and Performance-Optimized modes and can respond to out-of-memory errors by restarting tasks on larger VMs without manual intervention. Reported outcomes include a 99.998% upgrade success rate across more than 4.5 billion workloads, 2–5x faster Unilever pipelines, and operational-cost reductions of 25%.

Context

Traditional Spark deployments expose infrastructure directly to users, tightly coupling applications and compute. Shared-resource contention, cascading inefficiencies, changing demand, and manual performance-cost-reliability trade-offs make it difficult to maintain predictable execution and high utilization.

Approach / What changed

The architecture uses Spark Connect to separate user applications from compute infrastructure, a Serverless Gateway to route workloads using estimated query size, cluster utilization, and latency profile, and an adaptive autoscaler to adjust capacity horizontally and vertically. Workloads are isolated, placements are continuously re-evaluated, and out-of-memory tasks can restart on larger VMs without manual intervention.

Takeaways

• Spark Connect uses a client-server architecture over gRPC, shifts execution from application processes to queries, and lets the platform manage drivers independently of user workloads.
• The Serverless Gateway evaluates estimated query size, cluster-pool utilization, and interactive-versus-batch latency needs when routing workloads, then revisits placements as conditions change.
• The adaptive autoscaler provides Standard and Performance-Optimized modes, scales horizontally and vertically, and can restart an out-of-memory task on a larger VM so the job continues without manual intervention.

Related reading