Rethinking Distributed Systems for Serverless Performance and Reliability

Summary

Building serverless compute for Apache Spark requires more than warm machine pools or basic autoscaling because traditional deployments couple user applications to infrastructure, allowing contention and inefficiencies to undermine performance and reliability. The proposed architecture separates these concerns through Spark Connect’s client-server model over gRPC, a gateway that routes workloads using query size, cluster utilization, and latency profile, and an adaptive autoscaler that adjusts capacity horizontally and vertically. Spark Connect isolates user applications from drivers, while the gateway continually re-evaluates placement to reduce interference between workloads. The autoscaler offers Standard and Performance-Optimized modes and can respond to out-of-memory errors by restarting tasks on larger VMs without manual intervention. Reported outcomes include a 99.998% upgrade success rate across more than 4.5 billion workloads, 2–5x faster Unilever pipelines, and operational-cost reductions of 25%.

Takeaways

1. 1

   Spark Connect uses a client-server architecture over gRPC, shifts execution from application processes to queries, and lets the platform manage drivers independently of user workloads.

2. 2

   The Serverless Gateway evaluates estimated query size, cluster-pool utilization, and interactive-versus-batch latency needs when routing workloads, then revisits placements as conditions change.

3. 3

   The adaptive autoscaler provides Standard and Performance-Optimized modes, scales horizontally and vertically, and can restart an out-of-memory task on a larger VM so the job continues without manual intervention.