# Data First, SLA Always

[Grab](https://yomu.fyi/company/grab) · Johan Kok · Aug 1, 2019

**Type:** Problem & solution

## Summary

Grab's Data Engineering team transitioned from periodic batch ETL ingestion to a real-time change data capture architecture called Trailblazer after dataset sizes exceeded the petabyte mark. The previous system caused severe JDBC timeouts and heavy CPU loads when executing chunked or full-scan queries on unindexed upstream MySQL tables. To solve this, MySQL binary logs are captured via Debezium on Kafka Connect, buffered in Kafka, and ingested into a data lake using Spark Structured Streaming. Checkpoints are decoupled from local storage and persisted in a Redis cluster to simplify ingestion offset overrides and handle ephemeral compute clusters. The system incorporates extensive health monitoring across Airflow, Datadog, and custom services to maintain stream liveliness and avoid Kafka retention breaches.

## Context

Grab's batch ETL Loader application struggled as data surpassed petabyte scale, causing JDBC timeouts on growing tables and high CPU load when querying MySQL tables lacking proper update timestamps or indexes.

## Approach / What changed

Built Trailblazer, a change data capture pipeline that uses Debezium to stream MySQL binary logs into Kafka, consumes them via Spark Structured Streaming into a data lake, and manages checkpoints externally in Redis.

## Takeaways

- Externalizing Spark checkpoint management to Redis allows operators to alter starting offsets without modifying S3 metadata and prevents state loss when clusters terminate.
- Comparing ingested Kafka topic-partition offsets against total end offsets detects stream divergence before messages exceed Kafka's retention window.
- Airflow automates stream restarts and retries when transient infrastructure issues, such as DNS resolution failures or broker disk exhaustion, terminate Spark jobs.

**Tags:** [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Kafka](https://yomu.fyi/topic/kafka), [MySQL](https://yomu.fyi/topic/mysql), [Redis](https://yomu.fyi/topic/redis), [Streaming](https://yomu.fyi/topic/streaming)

[Read original post](https://engineering.grab.com/data-first-sla-always)