How Kafka Connect helps move data seamlessly

Summary

Grab's real-time data platform team, Coban, implemented a managed Kafka Connect ecosystem on Kubernetes to streamline moving data in and out of Apache Kafka. To resolve dual-write consistency issues and capture pre- and post-change data, the team integrated Debezium connectors to capture MySQL binlog events and accommodate database DDL migrations. For disaster recovery and stream migrations, Coban deployed MirrorMaker2 connectors managed via Terraform to handle message mirroring and consumer offset translation across AWS regions. Additionally, they developed a custom converter utilizing Confluent Schema Registry to transform Protobuf-serialized Kafka records into JSON for ingestion into Azure Event Hubs. This architecture enabled zero-downtime cluster migrations and robust cross-region disaster recovery.

Context

Grab needed to reliably move high-volume data in and out of Apache Kafka without dual-write consistency issues, while enabling disaster recovery with consumer offset translation and cross-cloud data ingestion into Azure Event Hubs.

Approach / What changed

The team adopted Kafka Connect on Kubernetes, deploying Debezium connectors for MySQL change data capture, deploying Kafka MirrorMaker2 connectors via Terraform for offset-aware cross-region mirroring, and developing a custom Protobuf-to-JSON runtime converter for Azure Event Hubs.

Takeaways

• Debezium connectors capture MySQL binlog changes alongside pre- and post-change snapshots, eliminating the need for upstream services to manage atomic dual writes to both MySQL and Kafka.
• Deploying MirrorMaker2 on Kafka Connect using Terraform allows granular control over MirrorSourceConnector and MirrorCheckpointConnector while enabling consumer offset translation during cross-region failovers.
• To bridge Protobuf-serialized Kafka streams with Azure Event Hubs, the team created a converter that deserializes messages into Protobuf DynamicMessage objects via Confluent Schema Registry and transforms field descriptors into JSON nodes using post-order depth-first search.

Related reading

• Grab · Apr 1, 2022

  Supporting large campaigns at scale

  Grab developed a batch job service within its Trident automation engine to execute multi-step marketing campaigns for millions of users simultaneously. The system replaces sequential, single-server execution with a distributed architecture powered by Apache Kafka, which distributes batches of 100 users across server clusters using hashed partition keys. To reduce network overhead and queries per second, downstream reward and messaging services introduced batch endpoints backed by bulk database queries, decreasing API latency by up to 85%. Grab further optimized performance by sharding Kafka topics by country and action type to prevent long-running reward tasks from blocking time-sensitive messaging workloads. Additionally, making terminal messaging calls asynchronous allows subsequent batch processing to proceed without waiting for message delivery confirmations.

  Jie Zhang

• Grab · Jul 7, 2021

  Reshaping Chat Support for Our Users

  Grab transitioned from voice hotlines and third-party tools to an in-house native chat support system integrated into their CRM. The team validated the platform through an MVP and user shadowing to address session disconnections, agent context switching, and routing bottlenecks. To optimize support operations at scale, they introduced dynamic queue limits based on Little's law, machine learning autocomplete suggestions for agents, and duration timers with visual nudges. These enhancements reduced chat waiting times by 30%, unresponsive users by 7%, and overall chat handling duration by 22%.

  Elisa Monacchi

• Grab · Jan 26, 2021

  Serving Driver-partners Data at Scale Using Mirror Cache

  Grab's Drivers Data service handles up to 10,000 requests per second during peak hours to supply driver information across backend microservices. The original setup used MySQL with Redis and standalone in-memory local caches, but yielded a low 25% local cache hit rate due to traffic patterns characterized by high burst frequency for individual drivers alongside redundant database calls across nodes. To solve this, the team developed Mirror Cache, an in-memory caching system that pairs Dgraph's Ristretto library with an asynchronous gRPC replication layer to mirror updates across cluster nodes. The replicator batches updates within the same AWS availability zone and forwards data to single nodes across zones to minimize transfer overhead. Production deployment increased the in-memory cache hit rate to approximately 75% and reduced direct MySQL queries by 5%.

  Indrajit Sarkar

• Grab · Jun 30, 2021

  Debugging High Latency Due to Context Leaks

  Market-Store, Grab's feature store for real-time machine learning features, experienced latency spikes from under 200 milliseconds to 2 seconds as traffic grew. Metrics and logs showed no direct correlation to API issues, but heap profiling with PPROF revealed continuously increasing memory held by child contexts. Further analysis tracked the leak to an update in Grab's open-source Async Library, which switched background contexts to uncancelled task contexts for worker runners. Because parent contexts maintained references to these uncancelled child contexts, the garbage collector could not reclaim their memory. This progressive memory exhaustion directly degraded API latency.

  Sourabh Suman