# How We Simplified Our Data Ingestion & Transformation Process

[Grab](https://yomu.fyi/company/grab) · Yichao Wang · Mar 3, 2019

**Type:** Problem & solution

## Summary

Grab evolved its real-time data ingestion pipeline after an initial architecture built on Spark Streaming and Python encountered operational complexity, node failures, and data loss from S3 eventual consistency. Because the streaming workload primarily handled event partitioning and ORC file generation, the team consolidated these tasks directly into an existing Golang processing service. They implemented sharded concurrent maps for high-throughput partitioning and optimized heap allocations to resolve memory bottlenecks. This refactor removed intermediate Avro conversions and intermediate storage hops. The simplified Go pipeline eliminated data loss and reduced processing lag from up to 13 minutes down to approximately 1 minute.

## Context

Grab's initial real-time data ingestion pipeline suffered from operational complexity on EMR, 4-to-13-minute processing lags, and data loss caused by Spark Streaming interacting with hourly S3 prefixes and eventual consistency at high volumes.

## Approach / What changed

Grab eliminated Spark Streaming, Python components, and intermediate Avro encoding by implementing event partitioning with sharded concurrent maps and direct Protobuf-to-ORC encoding within an optimized Golang transformation service.

## Takeaways

- Using Spark Streaming to consume objects from S3 caused data loss at over one terabyte per hour due to S3 eventual consistency and hourly prefix boundaries.
- Replacing the multi-stage pipeline simplified the conversion chain from Protobuf -> Avro -> ORC directly to Protobuf -> ORC.
- Using sharded concurrent maps in Golang coupled with heap allocation profiling allowed the service to maintain high write throughput and low read latency without memory exhaustion.

**Tags:** [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Go](https://yomu.fyi/topic/go), [Kafka](https://yomu.fyi/topic/kafka), [Scalability](https://yomu.fyi/topic/scalability), [Streaming](https://yomu.fyi/topic/streaming)

[Read original post](https://engineering.grab.com/data-ingestion-transformation-product-insights)