Loading…
Data Pipelines
65 posts about Data Pipelines. Every summary links to the original.
Grab ·
Automatic rule backtesting with large quantities of data
Evaluating new or modified risk rules previously required Grab analysts to run slow offline Presto queries, manually construct payloads, or run rules in shadow mode for days. To standardize and accelerate this workflow, Grab developed an automated backtesting system powered by an AWS EMR Spark pipeline. Historical events are continuously ingested via Kafka and a Kubernetes stream pipeline into S3 using Snappy-compressed Parquet. Users configure replay intervals and rule definitions directly in the rule engine UI, which triggers asynchronous Spark jobs through Amazon SQS and Lambda. This automated simulation replaces multi-week shadow mode runs and generates downloadable aggregation metrics on transactions, user counts, and treatment outcomes.
Chao WangGrab ·
How we store and process millions of orders daily
The Grab Order Platform processes millions of food and mart transactions daily, requiring high throughput, fault tolerance, and reduced cloud costs across transactional and analytical workloads. To meet these demands, the engineering team decoupled their database architecture by using Amazon DynamoDB for critical OLTP queries and MySQL RDS for historical OLAP queries. DynamoDB handles online order lifecycles with strong consistency, utilizing sparse Global Secondary Indexes for ongoing orders and TTL configurations to limit storage growth. Updates propagate asynchronously to MySQL RDS through a Kafka ingestion pipeline backed by Amazon SQS retries and timestamp-based version checks. This dual-database approach isolated core transaction availability from analytical queries and delivered significant cloud cost savings.
Xi ChenGrab ·
Graph concepts and applications
Real-world systems generate dynamic, non-random connections that traditional statistical approaches fail to characterize or forecast. Graph models represent these structures through vertices and edges, abstracting complex networks into mathematically tractable relationships. Common data representation formats include the Resource Description Framework (RDF), which models subject-predicate-object triples with IRIs, literals, and blank nodes, and Labeled Property Graphs (LPGs), which store arbitrary key-value properties directly on nodes and edges. Graph databases, derived from the LPG model, treat relationships with equal weight to entities, delivering responsive traversals for highly interconnected systems. While they offer agility and explicit relationship modeling, graph databases lack a standardized query language and remain poorly suited for standard transaction-focused workloads compared to relational databases.
Wenxiang LuGrab ·
Automated Experiment Analysis - Making experimental analysis scalable
Manual ad-hoc analysis of online controlled experiments at Grab introduced operational inefficiencies, inconsistent quality control, and scalability barriers across teams. To resolve these issues, Grab extended its GrabX experimentation platform with an Automated Experiment Analysis system that standardises metrics and automates statistical evaluations. The architecture stores experiment configurations and metric definitions from Cosmos DB into Azure Data Lake as bronze datasets, uses Spark on Databricks via Azure Data Factory to process subjects into silver datasets, and applies an internal Python Decision Engine to generate final gold results. These gold datasets are stored in star-schema fact and dimension tables and presented directly in the GrabX interface using embedded Power BI visualisations. The automation eliminates repetitive data pipeline construction for analysts, ensures reproducible findings aligned with initial hypotheses, and accelerates product launch decisions.
Albert ChengGrab ·
How Kafka Connect helps move data seamlessly
Grab's real-time data platform team, Coban, implemented a managed Kafka Connect ecosystem on Kubernetes to streamline moving data in and out of Apache Kafka. To resolve dual-write consistency issues and capture pre- and post-change data, the team integrated Debezium connectors to capture MySQL binlog events and accommodate database DDL migrations. For disaster recovery and stream migrations, Coban deployed MirrorMaker2 connectors managed via Terraform to handle message mirroring and consumer offset translation across AWS regions. Additionally, they developed a custom converter utilizing Confluent Schema Registry to transform Protobuf-serialized Kafka records into JSON for ingestion into Azure Event Hubs. This architecture enabled zero-downtime cluster migrations and robust cross-region disaster recovery.
Wenli WanGrab ·
Real-time data ingestion in Grab
Service teams at Grab historically had to dual-write transactional data into databases and Kafka, creating data integrity issues during transaction failures alongside substantial schema maintenance overhead. To overcome these limitations and eliminate burst reads from SQL-based queries, the Caspian team built a real-time ingestion platform synchronising MySQL, Aurora, and DynamoDB directly to Kafka. For MySQL and Aurora, the platform uses Debezium with Kafka Connect on ROW-format binlogs, while DynamoDB changes are captured via DynamoDB streams with auto-scaling AWS Lambda functions. Messages encoded in Protobuf are transported via Kafka and ingested into Amazon S3 using a Golang stream processor. This architecture supports search indexing in Elasticsearch, automated data lake pipelines, cross-region disaster recovery replication, and audit trails.
Shuguang XiangGrab ·
Abacus - Issuing points for multiple sources
Grab needed a centralised points management architecture to issue loyalty points across a growing catalog of products, membership tiers, and external partner exchanges. To address this, the engineering team built Abacus, an issuance platform designed to process millions of daily transactions with high availability. The system ingests completed transaction streams or API calls, dynamically computes points via configured multipliers, and passes calculations through Amazon Simple Queue Service queues. Once the Point Awarding module updates a persistent ledger, Abacus notifies consumers, emits events to Kafka for downstream consumers, and recalculates rolling point expiration dates.
ChandrakanthGrab ·
Automating Multi-Armed Bandit testing during feature rollout
Traditional feature rollouts and Multi-Armed Bandit testing operate as separate workflows that often depend on delayed offline analysis. To eliminate manual intervention, the Multi-Armed Bandit Optimiser automates testing concurrently during feature rollouts by responding to minute-level feedback metrics. The architecture connects Kafka Streams data processing, a metrics server with Spark jobs, and an adaptive rollout module updating online experimentation configurations. Candidate models are evaluated via Thompson Sampling on Beta distributions, with Monte Carlo simulations determining traffic allocation across user entities. In production for the GrabFood recommendation widget, the system optimizes the Effective Conversion Rate over a 30-minute window and includes fallback distribution logic.
Weicheng ZhuGrab ·
Processing ETL tasks with Ratchet
Grab's Lending team relies on multiple microservices to finance various user segments, requiring automated pipelines for operational tasks like loan rescheduling and merchant whitelisting. To execute these workflows, the team uses Ratchet, a Go library that organizes data pipelines into concurrent stages connected by Go channels and JSON payloads. In the Azkaban service, each ETL task is structured as a Job Processor that configures custom Data Processors for extraction, transformation, and loading. For merchant whitelisting, a custom processor pulls uploaded CSV files from Amazon S3, validates the rows to prevent manual errors, calls microservice APIs, and passes the output to an empty load stage. This pipeline architecture allows the team to isolate failures quickly, eliminate manual data validation, and automate routine operational tasks.
Amar PrakashGrab ·
Building a Hyper Self-Service, Distributed Tracing and Feedback System for Rule & Machine Learning (ML) Predictions
Grab's Trust, Identity, Safety, and Security team processes billions of daily rule and machine learning decisions for fraud detection, safety, and identity checks. Earlier logging approaches using plain text Kibana logs and the ActionTrace library lacked structured formats, dynamic entity customization, and fine-grained access controls. To resolve these limitations, the team built Archivist, a centralized tracing, statistics, and feedback system. Archivist ingests events through an SDK into Kafka streams, buffers and routes data into Elasticsearch indices and Amazon S3, and provides a role-based user portal. The platform handles 80 million daily logs across roughly 50 business scenarios, reducing scenario onboarding times from days to minutes.
Warren ZhouGrab ·
The Journey of Deploying Apache Airflow at Grab
Engineering and data teams across Grab originally operated independent Apache Airflow instances, causing duplicate maintenance overhead and frequent job failures around scaling, logging, and dependency management. To resolve this, a dedicated team developed a centralized orchestration platform that runs isolated, containerized Airflow instances per team on Amazon EKS. The platform categorizes deployments into three size tiers and provisions dedicated Redis brokers, RDS metadata stores, and Vault secret sidecars using Terraform and custom Helm charts. Teams customize container images using shared GitLab CI/CD templates, while worker scaling is handled via Kubernetes Horizontal Pod Autoscalers. Today, the platform runs roughly 20 Airflow instances executing between 1,000 and 60,000 daily jobs per instance.
Chandulal KavarGrab ·
Does Southeast Asia Run on Coffee?
GrabFood examined regional coffee ordering trends across major Southeast Asian cities over a nine-month period. Regional coffee orders expanded by 1,400%, with most countries recording their highest order volumes on Wednesdays before tapering off toward the weekend. Singapore and the Philippines deviated from this regional pattern, experiencing spikes in coffee orders on weekends and particularly on Sundays. Daily peak ordering times also differed across markets, peaking at 10:00 AM in Thailand, 2:00 PM in Indonesia, and 4:00 PM in Singapore. In addition to coffee, Green Tea Latte emerged as a top ten beverage item on the platform, accounting for over 25 million delivered cups.
Siu Sing LaiGrab ·
GrabChat Much? Talk Data to Me!
Grab launched GrabChat in September 2016 to facilitate in-app communication between passengers and driver-partners, subsequently adding features such as instant translation, image sharing, voice audio, and templates. An analysis of messaging data across Singapore, Malaysia, and Indonesia revealed distinct behavioral differences, with Indonesia averaging the highest volume at 5.5 chats per booking. Chat volume and the proportion of location-oriented messages peak between midnight and 4am across all three markets, driven by closed storefronts and obscure pickup spots. Image sharing sees higher utilization in remote areas, while pre-populated templates constitute nearly 50% of driver texts to reduce manual typing while driving. Furthermore, messaging responsiveness, directional guidance, and courtesy terms like saying thank you correlate directly with higher trip completion rates.
Jason Lee Jie ShienGrab ·
Data First, SLA Always
Grab's Data Engineering team transitioned from periodic batch ETL ingestion to a real-time change data capture architecture called Trailblazer after dataset sizes exceeded the petabyte mark. The previous system caused severe JDBC timeouts and heavy CPU loads when executing chunked or full-scan queries on unindexed upstream MySQL tables. To solve this, MySQL binary logs are captured via Debezium on Kafka Connect, buffered in Kafka, and ingested into a data lake using Spark Structured Streaming. Checkpoints are decoupled from local storage and persisted in a Redis cluster to simplify ingestion offset overrides and handle ephemeral compute clusters. The system incorporates extensive health monitoring across Airflow, Datadog, and custom services to maintain stream liveliness and avoid Kafka retention breaches.
Johan KokGrab ·
Tourists on GrabChat!
Grab examined more than 3.7 million tourist messages across Singapore, Malaysia, and Indonesia sent between December 2018 and March 2019 to evaluate passenger communication patterns. The platform deployed in-house translation and prewritten, auto-translated chat templates to bridge language barriers between international riders and local drivers. Analysis showed that bookings utilizing chat templates experienced a 10% higher ride completion rate than those without. Image-sharing features were most heavily used in high-traffic hubs such as airports, shopping malls, and major tourist centers to aid driver location. Passengers also consistently used messaging to clarify luggage capacity, provide identifiable passenger descriptions, and check pet policies.
Lara PuReum YimGrab ·
Bubble Tea Craze on GrabFood!
GrabFood recorded a regional average order growth rate of 3,000% for bubble tea across Southeast Asia in 2018. Individual country growth rates ranged from over 250% in Malaysia to more than 8,500% in Indonesia over their respective tracking periods. The customer base for bubble tea expanded by over 12,000%, supported by a 200% increase in merchant outlets to nearly 4,000 locations representing over 1,500 brands. Southeast Asian consumers ordered an average of four cups per person per month, led by Thailand at six cups and the Philippines at five cups. Order timing concentrated primarily around lunchtime meals and midday afternoon breaks.
Lara PuReum YimGrab ·
How We Harnessed the Wisdom of Crowds to Improve Restaurant Location Accuracy
Grab discovered that abnormally short driver wait times often indicated restaurants registered at incorrect coordinates due to moves or onboarding errors. To fix this, Grab used driver-partner GPS pings, timestamps, and order status updates to infer true food collection locations. The system cleans the data by filtering low-quality GPS pings and isolating the longest temporal streak a driver spends within a predefined radius of the venue. Clusters of inferred pick-up points are then ranked by order volume, the proportion of off-target pick-ups, and median distance errors before routing to mapping operations for verification. This periodic correction workflow achieved a fivefold reduction in order cancellations caused by unfound merchant locations.
Pravin KakarGrab ·
How We Simplified Our Data Ingestion & Transformation Process
Grab evolved its real-time data ingestion pipeline after an initial architecture built on Spark Streaming and Python encountered operational complexity, node failures, and data loss from S3 eventual consistency. Because the streaming workload primarily handled event partitioning and ORC file generation, the team consolidated these tasks directly into an existing Golang processing service. They implemented sharded concurrent maps for high-throughput partitioning and optimized heap allocations to resolve memory bottlenecks. This refactor removed intermediate Avro conversions and intermediate storage hops. The simplified Go pipeline eliminated data loss and reduced processing lag from up to 13 minutes down to approximately 1 minute.
Yichao WangGrab ·
Understanding Supply & Demand in Ride-hailing Through the Lens of Data
Grab measures ride-hailing supply and demand across space and time to resolve geo-temporal allocation mismatches between moving drivers and ride-seeking passengers. The analytics pipeline defines supply as idle online drivers and demand as passengers checking fares within brief time slots, aggregating locations into geohashes. Each driver is mapped across neighbouring demand units and inversely weighted by straight-line distance, which yields the effective supply, supply-demand ratio, and supply-demand difference for each geographic polygon. Grab uses these aggregated metrics to identify marketplace imbalances, deploying driver heatmaps to shift excess supply and passenger travel trend widgets to defer time-insensitive ride requests.
Aayush GargGrab ·
A Lean and Scalable Data Pipeline to Capture Large Scale Events and Support Experimentation Platform
Controlled online experimentation across diverse product verticals requires tracking interactions across systems to prevent local optimizations from causing global degradation. Grab built a batch data pipeline to capture, ingest, and process petabytes of event data to support its experimentation platform and analytics stakeholders. The architecture loads ingested event data from Amazon S3, transforms and sorts it, and writes partitioned output back to S3 with metadata registered in Apache Hive. Using Apache Spark on AWS Elastic MapReduce with Apache Airflow for orchestration, the system handles roughly 400,000 incoming events per second. The data is partitioned by event type and ingestion time and stored in Apache ORC format to streamline query workloads and reduce retrieval overhead.
Oscar Cassetti