# Kafka on Kubernetes: Reloaded for fault tolerance

[Grab](https://yomu.fyi/company/grab) · Fabrice Harbulot · Dec 26, 2023

**Type:** Problem & solution

## Summary

Grab's real-time data streaming platform, Coban, operates Kafka on AWS Elastic Kubernetes Service using Strimzi, allocating an entire EC2 worker node with NVMe instance store volumes to each broker. An initial architecture suffered from client connection errors, broken Network Load Balancer target groups, and zombie Persistent Volume Claims when worker nodes terminated. To achieve automated fault tolerance, the team integrated the AWS Node Termination Handler in Queue Processor mode with Auto Scaling lifecycle hooks, ensuring Kafka receives a SIGTERM to migrate partition leadership gracefully before shutdown. They also introduced the Kubernetes Cluster Autoscaler to dynamically provision replacement nodes during maintenance events and used the AWS Load Balancer Controller with TargetGroupBinding custom resources to dynamically update load balancer targets using IP mode.

## Context

Unexpected EC2 worker node terminations in a Kafka on Kubernetes setup caused client errors, stale Network Load Balancer target groups, and pod startup failures due to statically bound Persistent Volume Claims on deleted NVMe instance stores, requiring manual intervention.

## Approach / What changed

Deploy the AWS Node Termination Handler using SQS to drain nodes and trigger graceful Kafka leadership migration, pair Auto Scaling Groups with Kubernetes Cluster Autoscaler for node replacement, and use the AWS Load Balancer Controller with TargetGroupBinding CRDs for dynamic IP-based target group registration.

## Takeaways

- Setting Strimzi's terminationGracePeriodSeconds to 180 seconds provides sufficient time for Kafka to gracefully migrate partition leadership across surviving brokers upon receiving a SIGTERM.
- The AWS Node Termination Handler in Queue Processor mode consumes lifecycle and maintenance events from SQS to pause instance termination while cordoning and draining worker nodes.
- Using the AWS Load Balancer Controller with TargetGroupBinding custom resources switches NLB target groups to IP mode, automatically synchronising pod endpoints without manual reconfiguration.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [AWS](https://yomu.fyi/topic/aws), [Kafka](https://yomu.fyi/topic/kafka), [Kubernetes](https://yomu.fyi/topic/kubernetes), [Reliability](https://yomu.fyi/topic/reliability)

[Read original post](https://engineering.grab.com/kafka-on-kubernetes)