Data First, SLA Always

Summary

Grab's Data Engineering team transitioned from periodic batch ETL ingestion to a real-time change data capture architecture called Trailblazer after dataset sizes exceeded the petabyte mark. The previous system caused severe JDBC timeouts and heavy CPU loads when executing chunked or full-scan queries on unindexed upstream MySQL tables. To solve this, MySQL binary logs are captured via Debezium on Kafka Connect, buffered in Kafka, and ingested into a data lake using Spark Structured Streaming. Checkpoints are decoupled from local storage and persisted in a Redis cluster to simplify ingestion offset overrides and handle ephemeral compute clusters. The system incorporates extensive health monitoring across Airflow, Datadog, and custom services to maintain stream liveliness and avoid Kafka retention breaches.

Context

Grab's batch ETL Loader application struggled as data surpassed petabyte scale, causing JDBC timeouts on growing tables and high CPU load when querying MySQL tables lacking proper update timestamps or indexes.

Approach / What changed

Built Trailblazer, a change data capture pipeline that uses Debezium to stream MySQL binary logs into Kafka, consumes them via Spark Structured Streaming into a data lake, and manages checkpoints externally in Redis.

Takeaways

• Externalizing Spark checkpoint management to Redis allows operators to alter starting offsets without modifying S3 metadata and prevents state loss when clusters terminate.
• Comparing ingested Kafka topic-partition offsets against total end offsets detects stream divergence before messages exceed Kafka's retention window.
• Airflow automates stream restarts and retries when transient infrastructure issues, such as DNS resolution failures or broker disk exhaustion, terminate Spark jobs.

Related reading

• Grab · Mar 3, 2019

  How We Simplified Our Data Ingestion & Transformation Process

  Grab evolved its real-time data ingestion pipeline after an initial architecture built on Spark Streaming and Python encountered operational complexity, node failures, and data loss from S3 eventual consistency. Because the streaming workload primarily handled event partitioning and ORC file generation, the team consolidated these tasks directly into an existing Golang processing service. They implemented sharded concurrent maps for high-throughput partitioning and optimized heap allocations to resolve memory bottlenecks. This refactor removed intermediate Avro conversions and intermediate storage hops. The simplified Go pipeline eliminated data loss and reduced processing lag from up to 13 minutes down to approximately 1 minute.

  Yichao Wang

• Grab · Jan 16, 2019

  A Lean and Scalable Data Pipeline to Capture Large Scale Events and Support Experimentation Platform

  Controlled online experimentation across diverse product verticals requires tracking interactions across systems to prevent local optimizations from causing global degradation. Grab built a batch data pipeline to capture, ingest, and process petabytes of event data to support its experimentation platform and analytics stakeholders. The architecture loads ingested event data from Amazon S3, transforms and sorts it, and writes partitioned output back to S3 with metadata registered in Apache Hive. Using Apache Spark on AWS Elastic MapReduce with Apache Airflow for orchestration, the system handles roughly 400,000 incoming events per second. The data is partitioned by event type and ingestion time and stored in Apache ORC format to streamline query workloads and reduce retrieval overhead.

  Oscar Cassetti

• Grab · Jul 10, 2026

  Scaling Grab's Data Lake: Our journey to Apache Iceberg adoption

  Grab transitioned its petabyte-scale data lake from Hive Parquet to Apache Iceberg to resolve metadata bottlenecks, small file fragmentation, and lack of native ACID support. Implementing Iceberg with Z-ordering reduced navigation query runtimes from 70 seconds to 6 seconds and cut daily S3 API costs by up to 95% on key operational datasets. To seamlessly manage concurrent table formats without forcing engineers to specify format prefixes, Grab developed and open-sourced UnifiedSparkCatalog. This custom catalog transparently detects table types via the Hive Metastore, routes queries to format-specific handlers, and retains backwards compatibility for legacy Hive operations.

  Rahul Penti

• Grab · Aug 19, 2025

  Data mesh at Grab part I: Building trust through certification

  Rapid business growth across multiple verticals led Grab's centralized data engineering model to become an unscalable bottleneck, resulting in duplicate pipelines, ambiguous ownership, and broken downstream dependencies. To resolve these issues, the organization initiated a data mesh journey called Signals Marketplace that decentralizes data management and treats data as a product. A central data certification system establishes formal data contracts covering schemas, SLAs, freshness, and retention, while assigning clear Business Data Owners and Technical Data Owners. Breaches in contract guarantees automatically generate Data Production Incident tickets to enforce accountability and root-cause fixes. Consequently, 75% of internal queries now target certified assets, redundant tables saw a 400% year-over-year deprecation increase, and the total number of top-used datasets dropped by over 58%.

  Chun Rong Phang