The Hugo evolution: Engineering Grab's unified, one-click data ingestion platform with Apache Flink

Summary

Hugo evolved from a batch-oriented Spark platform into Grab’s unified, self-service data ingestion system for near real-time pipelines. The earlier expansion across Kafka Connect, Sprinkler, and Hugo fragmented onboarding, forcing users to translate configurations and manage manual schema changes across systems. A custom automation layer and Apache Flink now orchestrate one-click MySQL CDC and Kafka ingestion, retiring Kafka Connect and Sprinkler for the data lake. For CDC, Flink reads the binlog directly, uses automated schema detection and dynamic validation, and writes to queryable Hive tables through Spark compaction; for Kafka, it fetches Protobuf schemas from Confluent Schema Registry at runtime and resumes from the latest checkpoint after restart. Onboarding now takes approximately three minutes for MySQL CDC and six for Kafka, although schema updates still require manual restarts; new pipeline adoption in the last year also exceeded the previous five years combined.

Takeaways

1. 1

   Flink CDC reduced the MySQL ingestion footprint from four disparate components to two core components, removed the intermediary Kafka hop, and enabled direct binlog-to-Hive ingestion with Spark compaction.

2. 2

   Kafka pipelines dynamically retrieve Protobuf schemas from Confluent Schema Registry, but schema changes still require pipeline owners to manually restart Flink so processing can resume from the latest checkpoint.

3. 3

   Reported onboarding times are approximately six minutes for Kafka pipelines and three minutes for MySQL CDC pipelines; new pipeline adoption in the last year exceeded the total from the preceding five years.