How We Simplified Our Data Ingestion & Transformation Process

Summary

Grab evolved its real-time data ingestion pipeline after an initial architecture built on Spark Streaming and Python encountered operational complexity, node failures, and data loss from S3 eventual consistency. Because the streaming workload primarily handled event partitioning and ORC file generation, the team consolidated these tasks directly into an existing Golang processing service. They implemented sharded concurrent maps for high-throughput partitioning and optimized heap allocations to resolve memory bottlenecks. This refactor removed intermediate Avro conversions and intermediate storage hops. The simplified Go pipeline eliminated data loss and reduced processing lag from up to 13 minutes down to approximately 1 minute.

Takeaways

1. 1

   Using Spark Streaming to consume objects from S3 caused data loss at over one terabyte per hour due to S3 eventual consistency and hourly prefix boundaries.

2. 2

   Replacing the multi-stage pipeline simplified the conversion chain from Protobuf -> Avro -> ORC directly to Protobuf -> ORC.

3. 3

   Using sharded concurrent maps in Golang coupled with heap allocation profiling allowed the service to maintain high write throughput and low read latency without memory exhaustion.