# Data Pipelines > 78 posts about Data Pipelines, summarised, each linking to the original. ## Articles ### [Automating Multi-Armed Bandit testing during feature rollout](https://yomu.fyi/post/automating-multi-armed-bandit-testing-during-feature-rollout.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Weicheng Zhu - Published: Sep 1, 2021 Traditional feature rollouts and Multi-Armed Bandit testing operate as separate workflows that often depend on delayed offline analysis. To eliminate manual intervention, the Multi-Armed Bandit Optimiser automates testing concurrently during feature rollouts by responding to minute-level feedback metrics. The architecture connects Kafka Streams data processing, a metrics server with Spark jobs, and an adaptive rollout module updating online experimentation configurations. Candidate models are evaluated via Thompson Sampling on Beta distributions, with Monte Carlo simulations determining traffic allocation across user entities. In production for the GrabFood recommendation widget, the system optimizes the Effective Conversion Rate over a 30-minute window and includes fallback distribution logic. ### [Processing ETL tasks with Ratchet](https://yomu.fyi/post/processing-etl-tasks-with-ratchet.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Amar Prakash - Published: Jul 19, 2021 Grab's Lending team relies on multiple microservices to finance various user segments, requiring automated pipelines for operational tasks like loan rescheduling and merchant whitelisting. To execute these workflows, the team uses Ratchet, a Go library that organizes data pipelines into concurrent stages connected by Go channels and JSON payloads. In the Azkaban service, each ETL task is structured as a Job Processor that configures custom Data Processors for extraction, transformation, and loading. For merchant whitelisting, a custom processor pulls uploaded CSV files from Amazon S3, validates the rows to prevent manual errors, calls microservice APIs, and passes the output to an empty load stage. This pipeline architecture allows the team to isolate failures quickly, eliminate manual data validation, and automate routine operational tasks. ### [Building a Hyper Self-Service, Distributed Tracing and Feedback System for Rule & Machine Learning (ML) Predictions](https://yomu.fyi/post/building-a-hyper-self-service-distributed-tracing-and-feedback-system.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Warren Zhou - Published: May 24, 2021 Grab's Trust, Identity, Safety, and Security team processes billions of daily rule and machine learning decisions for fraud detection, safety, and identity checks. Earlier logging approaches using plain text Kibana logs and the ActionTrace library lacked structured formats, dynamic entity customization, and fine-grained access controls. To resolve these limitations, the team built Archivist, a centralized tracing, statistics, and feedback system. Archivist ingests events through an SDK into Kafka streams, buffers and routes data into Elasticsearch indices and Amazon S3, and provides a role-based user portal. The platform handles 80 million daily logs across roughly 50 business scenarios, reducing scenario onboarding times from days to minutes. ### [The Journey of Deploying Apache Airflow at Grab](https://yomu.fyi/post/the-journey-of-deploying-apache-airflow-at-grab.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Chandulal Kavar - Published: Jul 14, 2020 Engineering and data teams across Grab originally operated independent Apache Airflow instances, causing duplicate maintenance overhead and frequent job failures around scaling, logging, and dependency management. To resolve this, a dedicated team developed a centralized orchestration platform that runs isolated, containerized Airflow instances per team on Amazon EKS. The platform categorizes deployments into three size tiers and provisions dedicated Redis brokers, RDS metadata stores, and Vault secret sidecars using Terraform and custom Helm charts. Teams customize container images using shared GitLab CI/CD templates, while worker scaling is handled via Kubernetes Horizontal Pod Autoscalers. Today, the platform runs roughly 20 Airflow instances executing between 1,000 and 60,000 daily jobs per instance. ### [Does Southeast Asia Run on Coffee?](https://yomu.fyi/post/does-southeast-asia-run-on-coffee.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Siu Sing Lai - Published: Mar 26, 2020 GrabFood examined regional coffee ordering trends across major Southeast Asian cities over a nine-month period. Regional coffee orders expanded by 1,400%, with most countries recording their highest order volumes on Wednesdays before tapering off toward the weekend. Singapore and the Philippines deviated from this regional pattern, experiencing spikes in coffee orders on weekends and particularly on Sundays. Daily peak ordering times also differed across markets, peaking at 10:00 AM in Thailand, 2:00 PM in Indonesia, and 4:00 PM in Singapore. In addition to coffee, Green Tea Latte emerged as a top ten beverage item on the platform, accounting for over 25 million delivered cups. ### [GrabChat Much? Talk Data to Me!](https://yomu.fyi/post/grabchat-much-talk-data-to-me.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Jason Lee Jie Shien - Published: Mar 24, 2020 Grab launched GrabChat in September 2016 to facilitate in-app communication between passengers and driver-partners, subsequently adding features such as instant translation, image sharing, voice audio, and templates. An analysis of messaging data across Singapore, Malaysia, and Indonesia revealed distinct behavioral differences, with Indonesia averaging the highest volume at 5.5 chats per booking. Chat volume and the proportion of location-oriented messages peak between midnight and 4am across all three markets, driven by closed storefronts and obscure pickup spots. Image sharing sees higher utilization in remote areas, while pre-populated templates constitute nearly 50% of driver texts to reduce manual typing while driving. Furthermore, messaging responsiveness, directional guidance, and courtesy terms like saying thank you correlate directly with higher trip completion rates. ### [Data First, SLA Always](https://yomu.fyi/post/data-first-sla-always.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Johan Kok - Published: Aug 1, 2019 Grab's Data Engineering team transitioned from periodic batch ETL ingestion to a real-time change data capture architecture called Trailblazer after dataset sizes exceeded the petabyte mark. The previous system caused severe JDBC timeouts and heavy CPU loads when executing chunked or full-scan queries on unindexed upstream MySQL tables. To solve this, MySQL binary logs are captured via Debezium on Kafka Connect, buffered in Kafka, and ingested into a data lake using Spark Structured Streaming. Checkpoints are decoupled from local storage and persisted in a Redis cluster to simplify ingestion offset overrides and handle ephemeral compute clusters. The system incorporates extensive health monitoring across Airflow, Datadog, and custom services to maintain stream liveliness and avoid Kafka retention breaches. ### [Tourists on GrabChat!](https://yomu.fyi/post/tourists-on-grabchat.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Lara PuReum Yim - Published: May 22, 2019 Grab examined more than 3.7 million tourist messages across Singapore, Malaysia, and Indonesia sent between December 2018 and March 2019 to evaluate passenger communication patterns. The platform deployed in-house translation and prewritten, auto-translated chat templates to bridge language barriers between international riders and local drivers. Analysis showed that bookings utilizing chat templates experienced a 10% higher ride completion rate than those without. Image-sharing features were most heavily used in high-traffic hubs such as airports, shopping malls, and major tourist centers to aid driver location. Passengers also consistently used messaging to clarify luggage capacity, provide identifiable passenger descriptions, and check pet policies. ### [Bubble Tea Craze on GrabFood!](https://yomu.fyi/post/bubble-tea-craze-on-grabfood.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Lara PuReum Yim - Published: May 9, 2019 GrabFood recorded a regional average order growth rate of 3,000% for bubble tea across Southeast Asia in 2018. Individual country growth rates ranged from over 250% in Malaysia to more than 8,500% in Indonesia over their respective tracking periods. The customer base for bubble tea expanded by over 12,000%, supported by a 200% increase in merchant outlets to nearly 4,000 locations representing over 1,500 brands. Southeast Asian consumers ordered an average of four cups per person per month, led by Thailand at six cups and the Philippines at five cups. Order timing concentrated primarily around lunchtime meals and midday afternoon breaks. ### [How We Harnessed the Wisdom of Crowds to Improve Restaurant Location Accuracy](https://yomu.fyi/post/how-we-harnessed-the-wisdom-of-crowds-to-improve-restaurant-location-a.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Pravin Kakar - Published: Apr 2, 2019 Grab discovered that abnormally short driver wait times often indicated restaurants registered at incorrect coordinates due to moves or onboarding errors. To fix this, Grab used driver-partner GPS pings, timestamps, and order status updates to infer true food collection locations. The system cleans the data by filtering low-quality GPS pings and isolating the longest temporal streak a driver spends within a predefined radius of the venue. Clusters of inferred pick-up points are then ranked by order volume, the proportion of off-target pick-ups, and median distance errors before routing to mapping operations for verification. This periodic correction workflow achieved a fivefold reduction in order cancellations caused by unfound merchant locations. ### [How We Simplified Our Data Ingestion & Transformation Process](https://yomu.fyi/post/how-we-simplified-our-data-ingestion-transformation-process.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Yichao Wang - Published: Mar 3, 2019 Grab evolved its real-time data ingestion pipeline after an initial architecture built on Spark Streaming and Python encountered operational complexity, node failures, and data loss from S3 eventual consistency. Because the streaming workload primarily handled event partitioning and ORC file generation, the team consolidated these tasks directly into an existing Golang processing service. They implemented sharded concurrent maps for high-throughput partitioning and optimized heap allocations to resolve memory bottlenecks. This refactor removed intermediate Avro conversions and intermediate storage hops. The simplified Go pipeline eliminated data loss and reduced processing lag from up to 13 minutes down to approximately 1 minute. ### [Understanding Supply & Demand in Ride-hailing Through the Lens of Data](https://yomu.fyi/post/understanding-supply-demand-in-ride-hailing-through-the-lens-of-data.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Aayush Garg - Published: Feb 20, 2019 Grab measures ride-hailing supply and demand across space and time to resolve geo-temporal allocation mismatches between moving drivers and ride-seeking passengers. The analytics pipeline defines supply as idle online drivers and demand as passengers checking fares within brief time slots, aggregating locations into geohashes. Each driver is mapped across neighbouring demand units and inversely weighted by straight-line distance, which yields the effective supply, supply-demand ratio, and supply-demand difference for each geographic polygon. Grab uses these aggregated metrics to identify marketplace imbalances, deploying driver heatmaps to shift excess supply and passenger travel trend widgets to defer time-insensitive ride requests. ### [A Lean and Scalable Data Pipeline to Capture Large Scale Events and Support Experimentation Platform](https://yomu.fyi/post/a-lean-and-scalable-data-pipeline-to-capture-large-scale-events-and-su.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Oscar Cassetti - Published: Jan 16, 2019 Controlled online experimentation across diverse product verticals requires tracking interactions across systems to prevent local optimizations from causing global degradation. Grab built a batch data pipeline to capture, ingest, and process petabytes of event data to support its experimentation platform and analytics stakeholders. The architecture loads ingested event data from Amazon S3, transforms and sorts it, and writes partitioned output back to S3 with metadata registered in Apache Hive. Using Apache Spark on AWS Elastic MapReduce with Apache Airflow for orchestration, the system handles roughly 400,000 incoming events per second. The data is partitioned by event type and ingestion time and stored in Apache ORC format to streamline query workloads and reduce retrieval overhead. ### [Querying Big Data in Real-time with Presto & Grab's TalariaDB](https://yomu.fyi/post/querying-big-data-in-real-time-with-presto-grab-s-talariadb.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Roman Atachiants - Published: Jan 2, 2019 Grab developed TalariaDB to support real-time SQL querying over high-velocity event streams while maintaining predictable sub-second latencies and low infrastructure costs. The distributed time-series store retains only the most recent hour of data and integrates directly with Presto via its PrestoThriftService interface. Internally, TalariaDB uses the Go-based Badger key-value store to maintain an in-memory key index of metric names and timestamps while mapping columnar event payloads directly to disk. Ingestion occurs by processing pre-partitioned event batches written to Amazon S3 via SQS notifications. By combining a zero-copy decoder with parallel split evaluation across gossiping cluster nodes, the architecture scales horizontally while serving millions of events per second. ### [Journey of a Tourist via Grab](https://yomu.fyi/post/journey-of-a-tourist-via-grab.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Lara PuReum Yim - Published: Sep 11, 2018 Grab analyzed platform ride data from millions of tourist passengers representing over 150 countries visiting Singapore. Over 60% of these tourist riders originated from Southeast Asia, while non-regional visitors mainly came from China, the United States, and India. Seasonal demand revealed a trimodal distribution for tropical travelers aligning with holiday periods, contrasting with a September-to-January peak for visitors escaping winter in four-season climates. Airport trips showed that nearly 90% of tourist passengers headed directly to hotels, concentrated heavily in central areas like Orchard, Bugis, Downtown Core, and Kallang. Additional key destinations included major shopping districts, iconic dining locations like Newton Food Centre and Chijmes, and medical centers, which saw tourist ride volume grow over 500% between 2015 and 2017. ### [Building Grab’s Experimentation Platform](https://yomu.fyi/post/building-grab-s-experimentation-platform.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Abeesh Thomas - Published: Jul 13, 2018 Grab built its internal Experimentation Platform (ExP) to replace a manual, expensive testing process that required bespoke meetings, custom logging pipelines, and service modifications for each experiment. ExP provides a unified infrastructure featuring a centralized management UI, automated real-time data streaming to S3, and SDKs for Android, iOS, and Go. The platform leverages JSON-based experiment definitions delivered through dynamic configuration management, enabling client-side evaluation without costly network calls. It addresses marketplace network effects and inter-experiment interference through mechanisms such as geo-temporal segmentation and domain-layer models. The platform has scaled to run approximately 25 concurrent experiments while computing roughly 2,500 metrics and 50,000 experiment-metric combinations daily. ### [Driving Southeast Asia Forward with AWS](https://yomu.fyi/post/driving-southeast-asia-forward-with-aws.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Arul Kumaravel - Published: May 21, 2017 Grab transitioned its transportation platform from a single Ruby on Rails monolith on Amazon EC2 and Amazon RDS MySQL to a microservices architecture hosted on Amazon Web Services. The platform processes multi-petabyte real-time data flows and hundreds of millions of GPS data points to match drivers with passengers and push proactive demand heat maps. Operational efficiency is sustained with fewer than ten full-time infrastructure engineers, supported by AWS managed services. The analytics backend also transitioned from MySQL to Amazon Redshift, eventually moving to an Amazon S3 data lake using Amazon EMR and Presto. These data-driven matching systems improved driver-passenger allocation rates by up to 30%. ### [Scaling Like a Boss with Presto](https://yomu.fyi/post/scaling-like-a-boss-with-presto.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Aneesh Chandra - Published: May 1, 2017 Grab experienced severe performance degradation, long queue times, and connection timeouts on its Amazon Redshift analytics cluster as user concurrency and reporting workloads expanded. Although an initial Amazon S3 data lake decoupled storage from compute, business users required standard SQL interfaces rather than Spark data pipelines. The team deployed Presto clusters on AWS EMR, switching their storage format from AVRO to Parquet to support ANSI SQL querying directly against S3. Utilizing a shared Hive metastore on Amazon RDS allowed Grab to adopt a shared-data multi-cluster architecture that isolated distinct workloads across dedicated compute clusters. This setup enabled rapid cluster scaling, streamlined failover, and matched Redshift performance on partitioned time-range queries. [Newer posts](https://yomu.fyi/topic/data-pipelines/page/3.md)