Kafka on Kubernetes: Reloaded for fault tolerance

Summary

Grab's real-time data streaming platform, Coban, operates Kafka on AWS Elastic Kubernetes Service using Strimzi, allocating an entire EC2 worker node with NVMe instance store volumes to each broker. An initial architecture suffered from client connection errors, broken Network Load Balancer target groups, and zombie Persistent Volume Claims when worker nodes terminated. To achieve automated fault tolerance, the team integrated the AWS Node Termination Handler in Queue Processor mode with Auto Scaling lifecycle hooks, ensuring Kafka receives a SIGTERM to migrate partition leadership gracefully before shutdown. They also introduced the Kubernetes Cluster Autoscaler to dynamically provision replacement nodes during maintenance events and used the AWS Load Balancer Controller with TargetGroupBinding custom resources to dynamically update load balancer targets using IP mode.

Takeaways

1. 1

   Setting Strimzi's terminationGracePeriodSeconds to 180 seconds provides sufficient time for Kafka to gracefully migrate partition leadership across surviving brokers upon receiving a SIGTERM.

2. 2

   The AWS Node Termination Handler in Queue Processor mode consumes lifecycle and maintenance events from SQS to pause instance termination while cordoning and draining worker nodes.

3. 3

   Using the AWS Load Balancer Controller with TargetGroupBinding custom resources switches NLB target groups to IP mode, automatically synchronising pod endpoints without manual reconfiguration.