How Kafka Connect helps move data seamlessly

Summary

Grab's real-time data platform team, Coban, implemented a managed Kafka Connect ecosystem on Kubernetes to streamline moving data in and out of Apache Kafka. To resolve dual-write consistency issues and capture pre- and post-change data, the team integrated Debezium connectors to capture MySQL binlog events and accommodate database DDL migrations. For disaster recovery and stream migrations, Coban deployed MirrorMaker2 connectors managed via Terraform to handle message mirroring and consumer offset translation across AWS regions. Additionally, they developed a custom converter utilizing Confluent Schema Registry to transform Protobuf-serialized Kafka records into JSON for ingestion into Azure Event Hubs. This architecture enabled zero-downtime cluster migrations and robust cross-region disaster recovery.

Takeaways

1. 1

   Debezium connectors capture MySQL binlog changes alongside pre- and post-change snapshots, eliminating the need for upstream services to manage atomic dual writes to both MySQL and Kafka.

2. 2

   Deploying MirrorMaker2 on Kafka Connect using Terraform allows granular control over MirrorSourceConnector and MirrorCheckpointConnector while enabling consumer offset translation during cross-region failovers.

3. 3

   To bridge Protobuf-serialized Kafka streams with Azure Event Hubs, the team created a converter that deserializes messages into Protobuf DynamicMessage objects via Confluent Schema Registry and transforms field descriptors into JSON nodes using post-order depth-first search.