Optimally Scaling Kafka Consumer Applications

Summary

Grab's Coban platform runs Golang-based stream processing pipelines on Kubernetes, servicing roughly 400 billion events weekly from Kafka. The initial Horizontal Pod Autoscaler setup caused resource waste and uneven load distribution across Kafka partitions during scale-in and scale-out events. To resolve this, Grab moved to a fixed pod count matching the topic's partition count and adopted Vertical Pod Autoscaling, reducing resource usage versus requests by approximately 45%. The team also introduced Kubernetes priority classes to segment latency-sensitive workloads onto On-Demand nodes and non-critical jobs onto Spot instances. Additionally, overprovisioning via low-priority placeholder pods managed by Cluster Proportional Autoscaler enabled rapid pod rescheduling and reduced deployment delays.

Takeaways

1. 1

   Aligning the number of consumer pods directly with the Kafka topic partition count prevented uneven partition distribution during scaling events.

2. 2

   Replacing Horizontal Pod Autoscaling with Vertical Pod Autoscaling on fixed-size consumer deployments yielded an approximate 45% reduction in total resource usage relative to requested capacity.

3. 3

   Using low-priority placeholder pods scaled by Cluster Proportional Autoscaler allowed immediate preemption and rescheduling when Spot instances were terminated.