Real-time data ingestion in Grab

Summary

Service teams at Grab historically had to dual-write transactional data into databases and Kafka, creating data integrity issues during transaction failures alongside substantial schema maintenance overhead. To overcome these limitations and eliminate burst reads from SQL-based queries, the Caspian team built a real-time ingestion platform synchronising MySQL, Aurora, and DynamoDB directly to Kafka. For MySQL and Aurora, the platform uses Debezium with Kafka Connect on ROW-format binlogs, while DynamoDB changes are captured via DynamoDB streams with auto-scaling AWS Lambda functions. Messages encoded in Protobuf are transported via Kafka and ingested into Amazon S3 using a Golang stream processor. This architecture supports search indexing in Elasticsearch, automated data lake pipelines, cross-region disaster recovery replication, and audit trails.

Context

Service teams had to write transactional data twice (to Kafka and databases), risking data inconsistency on failed transactions, struggling with schema maintenance, and causing database read spikes from SQL-based pipeline queries.

Approach / What changed

Grab implemented an event-driven ingestion platform combining Debezium for MySQL and Aurora binlogs, AWS Lambda producers for DynamoDB streams, Kafka messaging with Protobuf encoding, and Golang stream processors dumping data to S3.

Takeaways

• Using Debezium integrated with Kafka Connect reduced database reads by 90 percent for the Grab Search indexing pipeline.
• AWS Lambda was chosen over a single-threaded Kafka Connect DynamoDB connector because Lambda dynamically auto-scales concurrency to handle partitioned DynamoDB stream surges.
• MySQL and Aurora binlogs preserve global ordering across records, whereas DynamoDB streams guarantee consistent ordering only within the same partition.

Related reading

• Grab · Jul 19, 2021

  Processing ETL tasks with Ratchet

  Grab's Lending team relies on multiple microservices to finance various user segments, requiring automated pipelines for operational tasks like loan rescheduling and merchant whitelisting. To execute these workflows, the team uses Ratchet, a Go library that organizes data pipelines into concurrent stages connected by Go channels and JSON payloads. In the Azkaban service, each ETL task is structured as a Job Processor that configures custom Data Processors for extraction, transformation, and loading. For merchant whitelisting, a custom processor pulls uploaded CSV files from Amazon S3, validates the rows to prevent manual errors, calls microservice APIs, and passes the output to an empty load stage. This pipeline architecture allows the team to isolate failures quickly, eliminate manual data validation, and automate routine operational tasks.

  Amar Prakash

• Grab · Oct 30, 2020

  Keeping 170 Libraries Up to Date on a Large Scale Android App

  Grab's passenger Android superapp relies on more than 170 in-house and open-source libraries, incorporating five to ten library bumps into each weekly release. Although developers often avoided updates due to the fear of leaking defects or crashes into production, the engineering team established a risk-assessment framework based on codebase usage and the volume of incoming changes. To minimize update diffs and avoid accumulating large issues, libraries are updated incrementally on a weekly cadence matching upstream release schedules. Automated UI test cases written in Gherkin syntax execute on CI for every merge request, while high-risk bumps trigger targeted QA manual testing. This structured updating process prevents defect leakage while keeping the application compliant with evolving Google Play target API requirements.

  Lucas Nelaupe

• Grab · Dec 22, 2020

  Pharos - Searching Nearby Drivers on Road Network at Scale

  Ride-hailing allocation requires identifying nearby drivers based on actual routing distance and estimated time of arrival rather than straight-line haversine distance. Grab developed Pharos, a distributed in-memory spatial microservice that evaluates road networks to match fast-moving drivers with pick-up requests. The system partitions OpenStreetMap road graphs by city and vehicle vertical while indexing driver positions and edge-based nodes using Adaptive Radix Trees. To locate candidates, Pharos projects driver coordinates onto road segments as phantom nodes and runs Incremental Network Expansion to compute isochrone reaches. Read-write concurrency is managed by switching tree root snapshots, allowing concurrent reads while updates produce new roots for subsequent lookups.

  Hao Wu

• Grab · Nov 22, 2021

  Using real-world patterns to improve matching in theory and practice

  Continuous ride-hailing assignment relies on solving the minimum weight bipartite matching problem between passengers and driver-partners. While traditional implementations assume a precalculated cost matrix, computing shortest-path travel times across large road networks dominates total execution time. Researchers introduced an Incremental Kuhn-Munkres algorithm that leverages the spatial locality of optimal matches to compute edge costs on demand. The approach integrates priority queues and lower-bounding techniques with refinement rules to avoid evaluating distant pairs while guaranteeing the same optimal assignment. Evaluated on Singapore road network data and real Grab production workloads, the incremental techniques reduced exact cost calculations and decreased assignment running times by over an order of magnitude.

  Tenindra Abeywickrama