# How Kafka Connect helps move data seamlessly

[Grab](https://yomu.fyi/company/grab) · Wenli Wan · Apr 6, 2022

**Type:** Problem & solution

## Summary

Grab's real-time data platform team, Coban, implemented a managed Kafka Connect ecosystem on Kubernetes to streamline moving data in and out of Apache Kafka. To resolve dual-write consistency issues and capture pre- and post-change data, the team integrated Debezium connectors to capture MySQL binlog events and accommodate database DDL migrations. For disaster recovery and stream migrations, Coban deployed MirrorMaker2 connectors managed via Terraform to handle message mirroring and consumer offset translation across AWS regions. Additionally, they developed a custom converter utilizing Confluent Schema Registry to transform Protobuf-serialized Kafka records into JSON for ingestion into Azure Event Hubs. This architecture enabled zero-downtime cluster migrations and robust cross-region disaster recovery.

## Context

Grab needed to reliably move high-volume data in and out of Apache Kafka without dual-write consistency issues, while enabling disaster recovery with consumer offset translation and cross-cloud data ingestion into Azure Event Hubs.

## Approach / What changed

The team adopted Kafka Connect on Kubernetes, deploying Debezium connectors for MySQL change data capture, deploying Kafka MirrorMaker2 connectors via Terraform for offset-aware cross-region mirroring, and developing a custom Protobuf-to-JSON runtime converter for Azure Event Hubs.

## Takeaways

- Debezium connectors capture MySQL binlog changes alongside pre- and post-change snapshots, eliminating the need for upstream services to manage atomic dual writes to both MySQL and Kafka.
- Deploying MirrorMaker2 on Kafka Connect using Terraform allows granular control over MirrorSourceConnector and MirrorCheckpointConnector while enabling consumer offset translation during cross-region failovers.
- To bridge Protobuf-serialized Kafka streams with Azure Event Hubs, the team created a converter that deserializes messages into Protobuf DynamicMessage objects via Confluent Schema Registry and transforms field descriptors into JSON nodes using post-order depth-first search.

**Tags:** [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Kafka](https://yomu.fyi/topic/kafka), [Kubernetes](https://yomu.fyi/topic/kubernetes), [MySQL](https://yomu.fyi/topic/mysql), [Streaming](https://yomu.fyi/topic/streaming)

[Read original post](https://engineering.grab.com/kafka-connect)