A Lean and Scalable Data Pipeline to Capture Large Scale Events and Support Experimentation Platform

Summary

Controlled online experimentation across diverse product verticals requires tracking interactions across systems to prevent local optimizations from causing global degradation. Grab built a batch data pipeline to capture, ingest, and process petabytes of event data to support its experimentation platform and analytics stakeholders. The architecture loads ingested event data from Amazon S3, transforms and sorts it, and writes partitioned output back to S3 with metadata registered in Apache Hive. Using Apache Spark on AWS Elastic MapReduce with Apache Airflow for orchestration, the system handles roughly 400,000 incoming events per second. The data is partitioned by event type and ingestion time and stored in Apache ORC format to streamline query workloads and reduce retrieval overhead.

Context

Rapid innovation and controlled experimentation across product verticals required tracking metrics and interactions across petabytes of event data, scaling to roughly 400,000 events per second while supporting diverse analytical SLAs with a lean engineering team.

Approach / What changed

Grab built a batch ETL pipeline using Apache Spark on AWS EMR, Apache Hive for the metastore, and Apache Airflow for orchestration, writing data to Amazon S3 in Apache ORC format partitioned hierarchically by event type and ingestion time.

Takeaways

• Partitioning data hierarchically by event type and ingestion time eliminated index overhead, enabled parallel processing of sub-partitions, and simplified storage-level access control.
• Benchmarking data storage formats showed Apache ORC with Snappy compression improved performance and storage utilization between 12.5% and 80% compared to Apache Parquet with Snappy.
• Because AWS EMR lacked hot-standby and Spark multi-master support, the team achieved high availability by deploying independent EMR clusters across availability zones with Airflow workers pulling jobs from a central queue.

Related reading

• Grab · Jul 13, 2018

  Building Grab’s Experimentation Platform

  Grab built its internal Experimentation Platform (ExP) to replace a manual, expensive testing process that required bespoke meetings, custom logging pipelines, and service modifications for each experiment. ExP provides a unified infrastructure featuring a centralized management UI, automated real-time data streaming to S3, and SDKs for Android, iOS, and Go. The platform leverages JSON-based experiment definitions delivered through dynamic configuration management, enabling client-side evaluation without costly network calls. It addresses marketplace network effects and inter-experiment interference through mechanisms such as geo-temporal segmentation and domain-layer models. The platform has scaled to run approximately 25 concurrent experiments while computing roughly 2,500 metrics and 50,000 experiment-metric combinations daily.

  Abeesh Thomas

• Grab · Aug 1, 2019

  Data First, SLA Always

  Grab's Data Engineering team transitioned from periodic batch ETL ingestion to a real-time change data capture architecture called Trailblazer after dataset sizes exceeded the petabyte mark. The previous system caused severe JDBC timeouts and heavy CPU loads when executing chunked or full-scan queries on unindexed upstream MySQL tables. To solve this, MySQL binary logs are captured via Debezium on Kafka Connect, buffered in Kafka, and ingested into a data lake using Spark Structured Streaming. Checkpoints are decoupled from local storage and persisted in a Redis cluster to simplify ingestion offset overrides and handle ephemeral compute clusters. The system incorporates extensive health monitoring across Airflow, Datadog, and custom services to maintain stream liveliness and avoid Kafka retention breaches.

  Johan Kok

• Grab · Mar 3, 2019

  How We Simplified Our Data Ingestion & Transformation Process

  Grab evolved its real-time data ingestion pipeline after an initial architecture built on Spark Streaming and Python encountered operational complexity, node failures, and data loss from S3 eventual consistency. Because the streaming workload primarily handled event partitioning and ORC file generation, the team consolidated these tasks directly into an existing Golang processing service. They implemented sharded concurrent maps for high-throughput partitioning and optimized heap allocations to resolve memory bottlenecks. This refactor removed intermediate Avro conversions and intermediate storage hops. The simplified Go pipeline eliminated data loss and reduced processing lag from up to 13 minutes down to approximately 1 minute.

  Yichao Wang

• Grab · May 1, 2017

  Scaling Like a Boss with Presto

  Grab experienced severe performance degradation, long queue times, and connection timeouts on its Amazon Redshift analytics cluster as user concurrency and reporting workloads expanded. Although an initial Amazon S3 data lake decoupled storage from compute, business users required standard SQL interfaces rather than Spark data pipelines. The team deployed Presto clusters on AWS EMR, switching their storage format from AVRO to Parquet to support ANSI SQL querying directly against S3. Utilizing a shared Hive metastore on Amazon RDS allowed Grab to adopt a shared-data multi-cluster architecture that isolated distinct workloads across dedicated compute clusters. This setup enabled rapid cluster scaling, streamlined failover, and matched Redshift performance on partitioned time-range queries.

  Aneesh Chandra