Data First, SLA Always

Summary

Grab's Data Engineering team transitioned from periodic batch ETL ingestion to a real-time change data capture architecture called Trailblazer after dataset sizes exceeded the petabyte mark. The previous system caused severe JDBC timeouts and heavy CPU loads when executing chunked or full-scan queries on unindexed upstream MySQL tables. To solve this, MySQL binary logs are captured via Debezium on Kafka Connect, buffered in Kafka, and ingested into a data lake using Spark Structured Streaming. Checkpoints are decoupled from local storage and persisted in a Redis cluster to simplify ingestion offset overrides and handle ephemeral compute clusters. The system incorporates extensive health monitoring across Airflow, Datadog, and custom services to maintain stream liveliness and avoid Kafka retention breaches.

Takeaways

1. 1

   Externalizing Spark checkpoint management to Redis allows operators to alter starting offsets without modifying S3 metadata and prevents state loss when clusters terminate.

2. 2

   Comparing ingested Kafka topic-partition offsets against total end offsets detects stream divergence before messages exceed Kafka's retention window.

3. 3

   Airflow automates stream restarts and retries when transient infrastructure issues, such as DNS resolution failures or broker disk exhaustion, terminate Spark jobs.