Balancing cost and reliability for Spark on Kubernetes

Summary

Notion describes how it reduced Spark compute costs on Kubernetes while maintaining reliability across production workloads. After moving from fixed EC2 clusters on Amazon EMR to EMR on EKS, it used Karpenter, EKS Auto Mode, and the MostAllocated scheduler for dynamic provisioning and bin-packing. Spot Instances then exposed a failure mode: one interruption could remove many bin-packed executors, causing shuffle loss and expensive recomputation. Spot Balancer addresses this by labeling executor creation through a Kubernetes webhook and maintaining each job’s configured spot-to-on-demand ratio, with per-job counts stored in Redis. Named stability weights make the cost-reliability trade-off explicit, and reported savings reached 60–90 percent while jobs that previously failed from interruptions finished reliably.

Takeaways

1. 1

   Spot interruptions were especially disruptive when many executors from one Spark job were bin-packed onto a single Spot Instance, potentially causing simultaneous executor loss and shuffle recomputation.

2. 2

   Spot Balancer lets each Spark job specify a target Spot-to-on-demand ratio, limiting the expected impact of interruptions while allowing longer or more critical jobs to use more stable capacity.

3. 3

   Notion reports 60–90 percent Spark compute cost savings after combining Kubernetes-based provisioning, bin-packing, and controlled Spot usage; previously failing jobs finished reliably.