Real-time data ingestion in Grab

Summary

Service teams at Grab historically had to dual-write transactional data into databases and Kafka, creating data integrity issues during transaction failures alongside substantial schema maintenance overhead. To overcome these limitations and eliminate burst reads from SQL-based queries, the Caspian team built a real-time ingestion platform synchronising MySQL, Aurora, and DynamoDB directly to Kafka. For MySQL and Aurora, the platform uses Debezium with Kafka Connect on ROW-format binlogs, while DynamoDB changes are captured via DynamoDB streams with auto-scaling AWS Lambda functions. Messages encoded in Protobuf are transported via Kafka and ingested into Amazon S3 using a Golang stream processor. This architecture supports search indexing in Elasticsearch, automated data lake pipelines, cross-region disaster recovery replication, and audit trails.

Takeaways

1. 1

   Using Debezium integrated with Kafka Connect reduced database reads by 90 percent for the Grab Search indexing pipeline.

2. 2

   AWS Lambda was chosen over a single-threaded Kafka Connect DynamoDB connector because Lambda dynamically auto-scales concurrency to handle partitioned DynamoDB stream surges.

3. 3

   MySQL and Aurora binlogs preserve global ordering across records, whereas DynamoDB streams guarantee consistent ordering only within the same partition.