# Data Pipelines
> 67 posts about Data Pipelines, summarised, each linking to the original.

## Articles

### [Query expansion based on user behaviour](https://yomu.fyi/post/query-expansion-based-on-user-behaviour.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Shuailong Liang
- Published: Nov 16, 2022

Grab developed a query expansion framework to resolve search issues such as typos, multi-language terms, missing alternatives, and word spacing. The corpus is generated through manual annotation for the top thousand fat-head queries and session-based data mining of user rewrite patterns. Offline pipelines extract adjacent query rewrites occurring within 30 seconds where only the second query generated a click, limiting candidates to three per query for serving efficiency. In production, expanded keywords pass through a query understanding, multi-recall, and result fusion flow with automated pruning of underperforming pairs. Online A/B testing across six Southeast Asian countries showed improvements in click-through and conversion rates, with data-mined expansions outperforming manual annotations alone.


### [Using mobile sensor data to encourage safer driving](https://yomu.fyi/post/using-mobile-sensor-data-to-encourage-safer-driving.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Laiyi Lin
- Published: Oct 25, 2022

Dedicated vehicular inertial measurement units are prohibitively expensive to install across large fleets, while customer feedback lacks the coverage required to assess driving safety comprehensively. To monitor driver behavior at scale, Grab processes smartphone sensor data collected via a mobile telematics SDK at high sampling frequencies. A custom orientation algorithm calculates Euler roll and pitch angles from gravity and resolves yaw through principal component analysis to align phone measurements with vehicle axes. The pipeline applies low-pass or Kalman filtering to handle varying noise levels, followed by threshold peak-finding algorithms to detect harsh acceleration, braking, and cornering. Speeding events are identified by comparing GPS speeds against known map limits or aggregated speed profiles derived from free-flowing traffic.


### [Automatic rule backtesting with large quantities of data](https://yomu.fyi/post/automatic-rule-backtesting-with-large-quantities-of-data.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Chao Wang
- Published: Sep 8, 2022

Evaluating new or modified risk rules previously required Grab analysts to run slow offline Presto queries, manually construct payloads, or run rules in shadow mode for days. To standardize and accelerate this workflow, Grab developed an automated backtesting system powered by an AWS EMR Spark pipeline. Historical events are continuously ingested via Kafka and a Kubernetes stream pipeline into S3 using Snappy-compressed Parquet. Users configure replay intervals and rule definitions directly in the rule engine UI, which triggers asynchronous Spark jobs through Amazon SQS and Lambda. This automated simulation replaces multi-week shadow mode runs and generates downloadable aggregation metrics on transactions, user counts, and treatment outcomes.


### [How we store and process millions of orders daily](https://yomu.fyi/post/how-we-store-and-process-millions-of-orders-daily.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Xi Chen
- Published: Aug 15, 2022

The Grab Order Platform processes millions of food and mart transactions daily, requiring high throughput, fault tolerance, and reduced cloud costs across transactional and analytical workloads. To meet these demands, the engineering team decoupled their database architecture by using Amazon DynamoDB for critical OLTP queries and MySQL RDS for historical OLAP queries. DynamoDB handles online order lifecycles with strong consistency, utilizing sparse Global Secondary Indexes for ongoing orders and TTL configurations to limit storage growth. Updates propagate asynchronously to MySQL RDS through a Kafka ingestion pipeline backed by Amazon SQS retries and timestamp-based version checks. This dual-database approach isolated core transaction availability from analytical queries and delivered significant cloud cost savings.


### [Graph concepts and applications](https://yomu.fyi/post/graph-concepts-and-applications.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Wenxiang Lu
- Published: Jun 2, 2022

Real-world systems generate dynamic, non-random connections that traditional statistical approaches fail to characterize or forecast. Graph models represent these structures through vertices and edges, abstracting complex networks into mathematically tractable relationships. Common data representation formats include the Resource Description Framework (RDF), which models subject-predicate-object triples with IRIs, literals, and blank nodes, and Labeled Property Graphs (LPGs), which store arbitrary key-value properties directly on nodes and edges. Graph databases, derived from the LPG model, treat relationships with equal weight to entities, delivering responsive traversals for highly interconnected systems. While they offer agility and explicit relationship modeling, graph databases lack a standardized query language and remain poorly suited for standard transaction-focused workloads compared to relational databases.


### [Automated Experiment Analysis - Making experimental analysis scalable](https://yomu.fyi/post/automated-experiment-analysis-making-experimental-analysis-scalable.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Albert Cheng
- Published: May 30, 2022

Manual ad-hoc analysis of online controlled experiments at Grab introduced operational inefficiencies, inconsistent quality control, and scalability barriers across teams. To resolve these issues, Grab extended its GrabX experimentation platform with an Automated Experiment Analysis system that standardises metrics and automates statistical evaluations. The architecture stores experiment configurations and metric definitions from Cosmos DB into Azure Data Lake as bronze datasets, uses Spark on Databricks via Azure Data Factory to process subjects into silver datasets, and applies an internal Python Decision Engine to generate final gold results. These gold datasets are stored in star-schema fact and dimension tables and presented directly in the GrabX interface using embedded Power BI visualisations. The automation eliminates repetitive data pipeline construction for analysts, ensures reproducible findings aligned with initial hypotheses, and accelerates product launch decisions.


### [How Kafka Connect helps move data seamlessly](https://yomu.fyi/post/how-kafka-connect-helps-move-data-seamlessly.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Wenli Wan
- Published: Apr 6, 2022

Grab's real-time data platform team, Coban, implemented a managed Kafka Connect ecosystem on Kubernetes to streamline moving data in and out of Apache Kafka. To resolve dual-write consistency issues and capture pre- and post-change data, the team integrated Debezium connectors to capture MySQL binlog events and accommodate database DDL migrations. For disaster recovery and stream migrations, Coban deployed MirrorMaker2 connectors managed via Terraform to handle message mirroring and consumer offset translation across AWS regions. Additionally, they developed a custom converter utilizing Confluent Schema Registry to transform Protobuf-serialized Kafka records into JSON for ingestion into Azure Event Hubs. This architecture enabled zero-downtime cluster migrations and robust cross-region disaster recovery.


### [Real-time data ingestion in Grab](https://yomu.fyi/post/real-time-data-ingestion-in-grab.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Shuguang Xiang
- Published: Mar 14, 2022

Service teams at Grab historically had to dual-write transactional data into databases and Kafka, creating data integrity issues during transaction failures alongside substantial schema maintenance overhead. To overcome these limitations and eliminate burst reads from SQL-based queries, the Caspian team built a real-time ingestion platform synchronising MySQL, Aurora, and DynamoDB directly to Kafka. For MySQL and Aurora, the platform uses Debezium with Kafka Connect on ROW-format binlogs, while DynamoDB changes are captured via DynamoDB streams with auto-scaling AWS Lambda functions. Messages encoded in Protobuf are transported via Kafka and ingested into Amazon S3 using a Golang stream processor. This architecture supports search indexing in Elasticsearch, automated data lake pipelines, cross-region disaster recovery replication, and audit trails.


### [Abacus - Issuing points for multiple sources](https://yomu.fyi/post/abacus-issuing-points-for-multiple-sources.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Chandrakanth
- Published: Mar 1, 2022

Grab needed a centralised points management architecture to issue loyalty points across a growing catalog of products, membership tiers, and external partner exchanges. To address this, the engineering team built Abacus, an issuance platform designed to process millions of daily transactions with high availability. The system ingests completed transaction streams or API calls, dynamically computes points via configured multipliers, and passes calculations through Amazon Simple Queue Service queues. Once the Point Awarding module updates a persistent ledger, Abacus notifies consumers, emits events to Kafka for downstream consumers, and recalculates rolling point expiration dates.


### [Automating Multi-Armed Bandit testing during feature rollout](https://yomu.fyi/post/automating-multi-armed-bandit-testing-during-feature-rollout.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Weicheng Zhu
- Published: Sep 1, 2021

Traditional feature rollouts and Multi-Armed Bandit testing operate as separate workflows that often depend on delayed offline analysis. To eliminate manual intervention, the Multi-Armed Bandit Optimiser automates testing concurrently during feature rollouts by responding to minute-level feedback metrics. The architecture connects Kafka Streams data processing, a metrics server with Spark jobs, and an adaptive rollout module updating online experimentation configurations. Candidate models are evaluated via Thompson Sampling on Beta distributions, with Monte Carlo simulations determining traffic allocation across user entities. In production for the GrabFood recommendation widget, the system optimizes the Effective Conversion Rate over a 30-minute window and includes fallback distribution logic.


### [Processing ETL tasks with Ratchet](https://yomu.fyi/post/processing-etl-tasks-with-ratchet.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Amar Prakash
- Published: Jul 19, 2021

Grab's Lending team relies on multiple microservices to finance various user segments, requiring automated pipelines for operational tasks like loan rescheduling and merchant whitelisting. To execute these workflows, the team uses Ratchet, a Go library that organizes data pipelines into concurrent stages connected by Go channels and JSON payloads. In the Azkaban service, each ETL task is structured as a Job Processor that configures custom Data Processors for extraction, transformation, and loading. For merchant whitelisting, a custom processor pulls uploaded CSV files from Amazon S3, validates the rows to prevent manual errors, calls microservice APIs, and passes the output to an empty load stage. This pipeline architecture allows the team to isolate failures quickly, eliminate manual data validation, and automate routine operational tasks.


### [Building a Hyper Self-Service, Distributed Tracing and Feedback System for Rule & Machine Learning (ML) Predictions](https://yomu.fyi/post/building-a-hyper-self-service-distributed-tracing-and-feedback-system.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Warren Zhou
- Published: May 24, 2021

Grab's Trust, Identity, Safety, and Security team processes billions of daily rule and machine learning decisions for fraud detection, safety, and identity checks. Earlier logging approaches using plain text Kibana logs and the ActionTrace library lacked structured formats, dynamic entity customization, and fine-grained access controls. To resolve these limitations, the team built Archivist, a centralized tracing, statistics, and feedback system. Archivist ingests events through an SDK into Kafka streams, buffers and routes data into Elasticsearch indices and Amazon S3, and provides a role-based user portal. The platform handles 80 million daily logs across roughly 50 business scenarios, reducing scenario onboarding times from days to minutes.


### [The Journey of Deploying Apache Airflow at Grab](https://yomu.fyi/post/the-journey-of-deploying-apache-airflow-at-grab.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Chandulal Kavar
- Published: Jul 14, 2020

Engineering and data teams across Grab originally operated independent Apache Airflow instances, causing duplicate maintenance overhead and frequent job failures around scaling, logging, and dependency management. To resolve this, a dedicated team developed a centralized orchestration platform that runs isolated, containerized Airflow instances per team on Amazon EKS. The platform categorizes deployments into three size tiers and provisions dedicated Redis brokers, RDS metadata stores, and Vault secret sidecars using Terraform and custom Helm charts. Teams customize container images using shared GitLab CI/CD templates, while worker scaling is handled via Kubernetes Horizontal Pod Autoscalers. Today, the platform runs roughly 20 Airflow instances executing between 1,000 and 60,000 daily jobs per instance.


### [Does Southeast Asia Run on Coffee?](https://yomu.fyi/post/does-southeast-asia-run-on-coffee.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Siu Sing Lai
- Published: Mar 26, 2020

GrabFood examined regional coffee ordering trends across major Southeast Asian cities over a nine-month period. Regional coffee orders expanded by 1,400%, with most countries recording their highest order volumes on Wednesdays before tapering off toward the weekend. Singapore and the Philippines deviated from this regional pattern, experiencing spikes in coffee orders on weekends and particularly on Sundays. Daily peak ordering times also differed across markets, peaking at 10:00 AM in Thailand, 2:00 PM in Indonesia, and 4:00 PM in Singapore. In addition to coffee, Green Tea Latte emerged as a top ten beverage item on the platform, accounting for over 25 million delivered cups.


### [GrabChat Much? Talk Data to Me!](https://yomu.fyi/post/grabchat-much-talk-data-to-me.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Jason Lee Jie Shien
- Published: Mar 24, 2020

Grab launched GrabChat in September 2016 to facilitate in-app communication between passengers and driver-partners, subsequently adding features such as instant translation, image sharing, voice audio, and templates. An analysis of messaging data across Singapore, Malaysia, and Indonesia revealed distinct behavioral differences, with Indonesia averaging the highest volume at 5.5 chats per booking. Chat volume and the proportion of location-oriented messages peak between midnight and 4am across all three markets, driven by closed storefronts and obscure pickup spots. Image sharing sees higher utilization in remote areas, while pre-populated templates constitute nearly 50% of driver texts to reduce manual typing while driving. Furthermore, messaging responsiveness, directional guidance, and courtesy terms like saying thank you correlate directly with higher trip completion rates.


### [Data First, SLA Always](https://yomu.fyi/post/data-first-sla-always.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Johan Kok
- Published: Aug 1, 2019

Grab's Data Engineering team transitioned from periodic batch ETL ingestion to a real-time change data capture architecture called Trailblazer after dataset sizes exceeded the petabyte mark. The previous system caused severe JDBC timeouts and heavy CPU loads when executing chunked or full-scan queries on unindexed upstream MySQL tables. To solve this, MySQL binary logs are captured via Debezium on Kafka Connect, buffered in Kafka, and ingested into a data lake using Spark Structured Streaming. Checkpoints are decoupled from local storage and persisted in a Redis cluster to simplify ingestion offset overrides and handle ephemeral compute clusters. The system incorporates extensive health monitoring across Airflow, Datadog, and custom services to maintain stream liveliness and avoid Kafka retention breaches.


### [Tourists on GrabChat!](https://yomu.fyi/post/tourists-on-grabchat.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Lara PuReum Yim
- Published: May 22, 2019

Grab examined more than 3.7 million tourist messages across Singapore, Malaysia, and Indonesia sent between December 2018 and March 2019 to evaluate passenger communication patterns. The platform deployed in-house translation and prewritten, auto-translated chat templates to bridge language barriers between international riders and local drivers. Analysis showed that bookings utilizing chat templates experienced a 10% higher ride completion rate than those without. Image-sharing features were most heavily used in high-traffic hubs such as airports, shopping malls, and major tourist centers to aid driver location. Passengers also consistently used messaging to clarify luggage capacity, provide identifiable passenger descriptions, and check pet policies.


### [Bubble Tea Craze on GrabFood!](https://yomu.fyi/post/bubble-tea-craze-on-grabfood.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Lara PuReum Yim
- Published: May 9, 2019

GrabFood recorded a regional average order growth rate of 3,000% for bubble tea across Southeast Asia in 2018. Individual country growth rates ranged from over 250% in Malaysia to more than 8,500% in Indonesia over their respective tracking periods. The customer base for bubble tea expanded by over 12,000%, supported by a 200% increase in merchant outlets to nearly 4,000 locations representing over 1,500 brands. Southeast Asian consumers ordered an average of four cups per person per month, led by Thailand at six cups and the Philippines at five cups. Order timing concentrated primarily around lunchtime meals and midday afternoon breaks.


### [How We Harnessed the Wisdom of Crowds to Improve Restaurant Location Accuracy](https://yomu.fyi/post/how-we-harnessed-the-wisdom-of-crowds-to-improve-restaurant-location-a.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Pravin Kakar
- Published: Apr 2, 2019

Grab discovered that abnormally short driver wait times often indicated restaurants registered at incorrect coordinates due to moves or onboarding errors. To fix this, Grab used driver-partner GPS pings, timestamps, and order status updates to infer true food collection locations. The system cleans the data by filtering low-quality GPS pings and isolating the longest temporal streak a driver spends within a predefined radius of the venue. Clusters of inferred pick-up points are then ranked by order volume, the proportion of off-target pick-ups, and median distance errors before routing to mapping operations for verification. This periodic correction workflow achieved a fivefold reduction in order cancellations caused by unfound merchant locations.


### [How We Simplified Our Data Ingestion & Transformation Process](https://yomu.fyi/post/how-we-simplified-our-data-ingestion-transformation-process.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Yichao Wang
- Published: Mar 3, 2019

Grab evolved its real-time data ingestion pipeline after an initial architecture built on Spark Streaming and Python encountered operational complexity, node failures, and data loss from S3 eventual consistency. Because the streaming workload primarily handled event partitioning and ORC file generation, the team consolidated these tasks directly into an existing Golang processing service. They implemented sharded concurrent maps for high-throughput partitioning and optimized heap allocations to resolve memory bottlenecks. This refactor removed intermediate Avro conversions and intermediate storage hops. The simplified Go pipeline eliminated data loss and reduced processing lag from up to 13 minutes down to approximately 1 minute.


[Newer posts](https://yomu.fyi/topic/data-pipelines/page/2.md) · [Older posts](https://yomu.fyi/topic/data-pipelines/page/4.md)