Kafka on Kubernetes: Reloaded for fault tolerance

Summary

Grab's real-time data streaming platform, Coban, operates Kafka on AWS Elastic Kubernetes Service using Strimzi, allocating an entire EC2 worker node with NVMe instance store volumes to each broker. An initial architecture suffered from client connection errors, broken Network Load Balancer target groups, and zombie Persistent Volume Claims when worker nodes terminated. To achieve automated fault tolerance, the team integrated the AWS Node Termination Handler in Queue Processor mode with Auto Scaling lifecycle hooks, ensuring Kafka receives a SIGTERM to migrate partition leadership gracefully before shutdown. They also introduced the Kubernetes Cluster Autoscaler to dynamically provision replacement nodes during maintenance events and used the AWS Load Balancer Controller with TargetGroupBinding custom resources to dynamically update load balancer targets using IP mode.

Context

Unexpected EC2 worker node terminations in a Kafka on Kubernetes setup caused client errors, stale Network Load Balancer target groups, and pod startup failures due to statically bound Persistent Volume Claims on deleted NVMe instance stores, requiring manual intervention.

Approach / What changed

Deploy the AWS Node Termination Handler using SQS to drain nodes and trigger graceful Kafka leadership migration, pair Auto Scaling Groups with Kubernetes Cluster Autoscaler for node replacement, and use the AWS Load Balancer Controller with TargetGroupBinding CRDs for dynamic IP-based target group registration.

Takeaways

• Setting Strimzi's terminationGracePeriodSeconds to 180 seconds provides sufficient time for Kafka to gracefully migrate partition leadership across surviving brokers upon receiving a SIGTERM.
• The AWS Node Termination Handler in Queue Processor mode consumes lifecycle and maintenance events from SQS to pause instance termination while cordoning and draining worker nodes.
• Using the AWS Load Balancer Controller with TargetGroupBinding custom resources switches NLB target groups to IP mode, automatically synchronising pod endpoints without manual reconfiguration.

Related reading

• Grab · Aug 15, 2023

  Streamlining Grab's Segmentation Platform with faster creation and lower latency

  Grab's Segmentation Platform previously stored user-to-segment mappings across individual rows in ScyllaDB, causing write bottlenecks during segment creation and read latencies too high for downstream consumers. To resolve these performance limitations, the team transitioned to storing segments as Roaring Bitmaps saved as single blobs in object storage. This compression strategy splits 32-bit integer user IDs into chunks across array, bitmap, and run containers based on data density, reducing a one-million-member segment to under one megabyte. An accompanying client SDK manages segment retrieval, decoding, update notifications, and least-recently-used in-memory caching. Consequently, consumers such as Grab's communications platform achieved peak throughput of 15,000 queries per second with sub-millisecond p99 read latencies.

  Jake Ng

• Grab · Aug 5, 2024

  How we improved translation experience with cost efficiency

  Grab faced translation quality and cost challenges in its booking chat system when tourism resumed after COVID restrictions eased. Device language settings proved unreliable for determining message language, prompting the team to use Lingua alongside booking context heuristics and third-party fallbacks for real-time detection. To replace expensive and inaccurate general translation APIs, Grab distilled a large open-source Hugging Face model into lightweight, language-specific models trained on ten million synthetic chat examples. A post-translation validation step verifies non-translatable entities such as numbers and emojis before falling back to external services if mutations occur. Caching layers were also introduced across translation paths to curb redundant on-the-fly execution.

  Jie Zhang

• Grab · Jul 15, 2024

  LLM-powered data classification for data entities at scale

  Grab needed to classify sensitive data at the table and column level across petabytes of database tables and streaming schemas. Manual schema-tiering campaigns had resulted in half of all schemas receiving overly strict Tier 1 access controls, while an initial automated service using regex patterns and third-party machine learning produced high false-positive rates and lacked customizability. To address this, the Caspian data engineering and governance teams enhanced their internal orchestration service, Gemini, by integrating GPT-3.5 via Azure OpenAI. Gemini aggregates classification requests into mini-batches, handles API rate limits, and uses prompt engineering—including few-shot examples, curated tag libraries, and explicit JSON DTO schemas—to reliably tag columns for data owner verification.

  Hualin Liu

• Grab · Aug 15, 2022

  How we store and process millions of orders daily

  The Grab Order Platform processes millions of food and mart transactions daily, requiring high throughput, fault tolerance, and reduced cloud costs across transactional and analytical workloads. To meet these demands, the engineering team decoupled their database architecture by using Amazon DynamoDB for critical OLTP queries and MySQL RDS for historical OLAP queries. DynamoDB handles online order lifecycles with strong consistency, utilizing sparse Global Secondary Indexes for ongoing orders and TTL configurations to limit storage growth. Updates propagate asynchronously to MySQL RDS through a Kafka ingestion pipeline backed by Amazon SQS retries and timestamp-based version checks. This dual-database approach isolated core transaction availability from analytical queries and delivered significant cloud cost savings.

  Xi Chen