# A Lean and Scalable Data Pipeline to Capture Large Scale Events and Support Experimentation Platform

[Grab](https://yomu.fyi/company/grab) · Oscar Cassetti · Jan 16, 2019

**Type:** Problem & solution

## Summary

Controlled online experimentation across diverse product verticals requires tracking interactions across systems to prevent local optimizations from causing global degradation. Grab built a batch data pipeline to capture, ingest, and process petabytes of event data to support its experimentation platform and analytics stakeholders. The architecture loads ingested event data from Amazon S3, transforms and sorts it, and writes partitioned output back to S3 with metadata registered in Apache Hive. Using Apache Spark on AWS Elastic MapReduce with Apache Airflow for orchestration, the system handles roughly 400,000 incoming events per second. The data is partitioned by event type and ingestion time and stored in Apache ORC format to streamline query workloads and reduce retrieval overhead.

## Context

Rapid innovation and controlled experimentation across product verticals required tracking metrics and interactions across petabytes of event data, scaling to roughly 400,000 events per second while supporting diverse analytical SLAs with a lean engineering team.

## Approach / What changed

Grab built a batch ETL pipeline using Apache Spark on AWS EMR, Apache Hive for the metastore, and Apache Airflow for orchestration, writing data to Amazon S3 in Apache ORC format partitioned hierarchically by event type and ingestion time.

## Takeaways

- Partitioning data hierarchically by event type and ingestion time eliminated index overhead, enabled parallel processing of sub-partitions, and simplified storage-level access control.
- Benchmarking data storage formats showed Apache ORC with Snappy compression improved performance and storage utilization between 12.5% and 80% compared to Apache Parquet with Snappy.
- Because AWS EMR lacked hot-standby and Spark multi-master support, the team achieved high availability by deploying independent EMR clusters across availability zones with Airflow workers pulling jobs from a central queue.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [AWS](https://yomu.fyi/topic/aws), [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Performance](https://yomu.fyi/topic/performance), [Scalability](https://yomu.fyi/topic/scalability)

[Read original post](https://engineering.grab.com/experimentation-platform-data-pipeline)