# Real-time data ingestion in Grab

[Grab](https://yomu.fyi/company/grab) · Shuguang Xiang · Mar 14, 2022

**Type:** Problem & solution

## Summary

Service teams at Grab historically had to dual-write transactional data into databases and Kafka, creating data integrity issues during transaction failures alongside substantial schema maintenance overhead. To overcome these limitations and eliminate burst reads from SQL-based queries, the Caspian team built a real-time ingestion platform synchronising MySQL, Aurora, and DynamoDB directly to Kafka. For MySQL and Aurora, the platform uses Debezium with Kafka Connect on ROW-format binlogs, while DynamoDB changes are captured via DynamoDB streams with auto-scaling AWS Lambda functions. Messages encoded in Protobuf are transported via Kafka and ingested into Amazon S3 using a Golang stream processor. This architecture supports search indexing in Elasticsearch, automated data lake pipelines, cross-region disaster recovery replication, and audit trails.

## Context

Service teams had to write transactional data twice (to Kafka and databases), risking data inconsistency on failed transactions, struggling with schema maintenance, and causing database read spikes from SQL-based pipeline queries.

## Approach / What changed

Grab implemented an event-driven ingestion platform combining Debezium for MySQL and Aurora binlogs, AWS Lambda producers for DynamoDB streams, Kafka messaging with Protobuf encoding, and Golang stream processors dumping data to S3.

## Takeaways

- Using Debezium integrated with Kafka Connect reduced database reads by 90 percent for the Grab Search indexing pipeline.
- AWS Lambda was chosen over a single-threaded Kafka Connect DynamoDB connector because Lambda dynamically auto-scales concurrency to handle partitioned DynamoDB stream surges.
- MySQL and Aurora binlogs preserve global ordering across records, whereas DynamoDB streams guarantee consistent ordering only within the same partition.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Kafka](https://yomu.fyi/topic/kafka), [MySQL](https://yomu.fyi/topic/mysql), [Streaming](https://yomu.fyi/topic/streaming)

[Read original post](https://engineering.grab.com/real-time-data-ingestion)