---
title: "Data Pipelines"
description: "248 posts about Data Pipelines, summarised, each linking to the original."
---

# Data Pipelines
> 248 posts about Data Pipelines, summarised, each linking to the original.

## Articles

### [Shopify's Playbook for Scaling Machine Learning](https://yomu.fyi/post/shopify-s-playbook-for-scaling-machine-learning.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Feb 11, 2022
- Type: Explainer

Shopify describes how its machine-learning practice grew from a first order-fraud product into a portfolio used across products such as Shopify Capital, product categorization, and Help Center search. The playbook starts with choosing a user-important problem, validating trustworthy and accessible data, tracing downstream dependencies, understanding existing solutions, and optimizing for product outcomes rather than model scores alone. To productionize models, Shopify recommends well-defined training, verification, and testing pipelines, deployment decisions based on volume and user commitment, and platform practices that encode validated standards without abstracting too early. Its fraud pipeline builds models in Python, serializes them with PMML for deployment in Ruby-based production systems, and uses Apache Airflow to schedule operations. The account concludes that scaling is a full-stack problem requiring data scientists and data engineers to align on one roadmap and goal.


### [Shopify’s Unique Data Science Hierarchy Of Needs](https://yomu.fyi/post/shopify-s-unique-data-science-hierarchy-of-needs.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 21, 2021
- Type: Explainer

Shopify presents a five-level Data Science Hierarchy of Needs that organizes data work by the value it delivers, rather than by technical sophistication. Its tool-agnostic philosophy favors simple solutions first, then more complex analysis when necessary, with impact and better decisions as the goal. The model begins with collecting and modeling clean, conformed data using dimensional modeling, then moves through description, prediction and inference, prescription, and influence. Shopify applied this structure to a COVID-19 impact analysis, using statistical analysis to examine regional effects and inform measures including an extended free trial, expanded funding, shipping options, and local commerce features. The account says the analysis used neither artificial intelligence nor machine learning, yet supported decisions for merchants and Shopify during the pandemic.


### [Building a Real-time Buyer Signal Data Pipeline for Shopify Inbox](https://yomu.fyi/post/building-a-real-time-buyer-signal-data-pipeline-for-shopify-inbox.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 13, 2021
- Type: Problem & solution

Shopify built a real-time buyer-signal pipeline for Shopify Inbox after noting that 70 percent of its conversations involved customers making a purchasing decision. The system surfaces cart actions and order-completion information as chronological conversation events, covering events from up to 14 days before a conversation and post-conversation events while its seven-day lifecycle remains active. It uses Kafka through Monorail and Change Data Capture, Apache Beam on Google Cloud Dataflow, filtering jobs for mission-critical transactional events, and stateful aggregation with timers and a Global Window to retain buyer activity. In an equal-group A/B test, buyer signals increased merchant response rate by two percentage points and attributed conversion rate by 0.7 percentage points, while response time showed no significant change.


### [Automating Multi-Armed Bandit testing during feature rollout](https://yomu.fyi/post/automating-multi-armed-bandit-testing-during-feature-rollout.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Weicheng Zhu
- Published: Sep 1, 2021
- Type: Problem & solution

Traditional feature rollouts and Multi-Armed Bandit testing operate as separate workflows that often depend on delayed offline analysis. To eliminate manual intervention, the Multi-Armed Bandit Optimiser automates testing concurrently during feature rollouts by responding to minute-level feedback metrics. The architecture connects Kafka Streams data processing, a metrics server with Spark jobs, and an adaptive rollout module updating online experimentation configurations. Candidate models are evaluated via Thompson Sampling on Beta distributions, with Monte Carlo simulations determining traffic allocation across user entities. In production for the GrabFood recommendation widget, the system optimizes the Effective Conversion Rate over a 30-minute window and includes fallback distribution logic.


### [Processing ETL tasks with Ratchet](https://yomu.fyi/post/processing-etl-tasks-with-ratchet.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Amar Prakash
- Published: Jul 19, 2021
- Type: Explainer

Grab's Lending team relies on multiple microservices to finance various user segments, requiring automated pipelines for operational tasks like loan rescheduling and merchant whitelisting. To execute these workflows, the team uses Ratchet, a Go library that organizes data pipelines into concurrent stages connected by Go channels and JSON payloads. In the Azkaban service, each ETL task is structured as a Job Processor that configures custom Data Processors for extraction, transformation, and loading. For merchant whitelisting, a custom processor pulls uploaded CSV files from Amazon S3, validates the rows to prevent manual errors, calls microservice APIs, and passes the output to an empty load stage. This pipeline architecture allows the team to isolate failures quickly, eliminate manual data validation, and automate routine operational tasks.


### [Deleting the Undeletable: Managing PII at Shopify - Shopify](https://yomu.fyi/post/deleting-the-undeletable-managing-pii-at-shopify-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 10, 2021
- Type: Problem & solution

Shopify describes how its Privacy and Data Science & Engineering teams redesigned analytical event handling to make personal-information deletion workable at Shopify’s scale. The effort addressed a lack of guaranteed event schemas by introducing a human-readable JSON schematization system covering field types, version evolution, ownership, data controllers and subjects, and privacy instructions; more than 4,500 active schemas now govern events. Before warehouse storage, identifying data is either obfuscated into generalized values, such as masked IP addresses with city or country information, or tokenized into consistent random values whose mappings reside in a secured vault and vary by controller and subject. Removing a vault mapping makes warehouse copies of the token non-detokenizable, allowing deletion without warehouse operations; the production platform processes about 20 billion events daily, while tens to hundreds of millions of mappings are deleted daily.


### [How Shopify Built An In-Context Analytics Experience](https://yomu.fyi/post/how-shopify-built-an-in-context-analytics-experience.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: May 25, 2021
- Type: Explainer

Shopify’s Analytics section gave merchants access to business data, but research and quantitative analysis found that they often moved between it and admin pages while making decisions. The team therefore designed an in-context experience for the Orders page, selecting metrics through workflow analysis, segmenting merchants by factors such as order volume and fulfillment process, and examining each metric’s usefulness, variability, and time horizon. Data scientists prototyped visualizations with real, segmented data and created a specification sheet so Web, Android, and iOS used consistent queries, data sources, fields, and aggregations; the project also tested batch models before streaming them. The launched experience surfaces order and fulfillment metrics for daily, seven-day, and thirty-day periods, with comparison indicators, and links to reports covering orders, returns, products, and fulfillment, shipping, and delivery times.


### [Building a Hyper Self-Service, Distributed Tracing and Feedback System for Rule & Machine Learning (ML) Predictions](https://yomu.fyi/post/building-a-hyper-self-service-distributed-tracing-and-feedback-system.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Warren Zhou
- Published: May 24, 2021
- Type: Problem & solution

Grab's Trust, Identity, Safety, and Security team processes billions of daily rule and machine learning decisions for fraud detection, safety, and identity checks. Earlier logging approaches using plain text Kibana logs and the ActionTrace library lacked structured formats, dynamic entity customization, and fine-grained access controls. To resolve these limitations, the team built Archivist, a centralized tracing, statistics, and feedback system. Archivist ingests events through an SDK into Kafka streams, buffers and routes data into Elasticsearch indices and Amazon S3, and provides a role-based user portal. The platform handles 80 million daily logs across roughly 50 business scenarios, reducing scenario onboarding times from days to minutes.


### [Capturing Every Change From Shopify’s Sharded Monolith](https://yomu.fyi/post/capturing-every-change-from-shopify-s-sharded-monolith.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Mar 12, 2021
- Type: Problem & solution

Shopify’s separate internal warehouse and Merchant Analytics Platform created duplicated work, differing extraction results, and incompatible tooling while demand grew for fresher, low-latency merchant data. Longboat, its query-based CDC service, polls sharded-monolith replicas in batch jobs using updated\_at, but queries can run only hourly and cannot detect hard deletes or updates that bypass that field. Shopify therefore adopted Kafka and Debezium CDC to capture database changes, standardize extraction, and make the event stream usable by both streaming and batch consumers. CDC-backed Beam modeling reduced average freshness for one migration from one day to one hour, while Kafka became a shared source for operational and analytical data. Shopify still identifies incremental snapshotting and streaming-state problems, including reconciling MySQL table chunks with ongoing binlog events without expensive locks, as ongoing work.


### [How to Reliably Scale Your Data Platform for High Volumes](https://yomu.fyi/post/how-to-reliably-scale-your-data-platform-for-high-volumes.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 8, 2020
- Type: Problem & solution

Shopify’s data platform prepared for BFCM 2020, when merchant sales exceeded $5.1 billion and platform throughput rose by an average of 150 percent. The Data Platform Engineering team focused on throughput risk, which grows with volume and is greatest near ingestion and processing, while prioritizing work through four service tiers. For Tier 1 and Tier 2 systems, engineers defined each service’s primary objective, identified adjustable controls such as job frequency, overlap intervals, CPU and memory requests, BigTable node pools, and ProxySQL connection queues, then used controlled load tests to expose capacity and recovery risks. The resulting work included cluster scaling, load-balancing adjustments, query and job tuning, and improved indices, schemas, dashboards, playbooks, SLOs, incident handling, and alerts. The post reports that BFCM 2020 completed without merchant-impacting incidents; only a handful of minor, self-healing incidents occurred, while monitoring and occasional tuning preserved spare capacity.


### [How to Build an Experiment Pipeline from Scratch](https://yomu.fyi/post/how-to-build-an-experiment-pipeline-from-scratch.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 15, 2020
- Type: Tutorial

The post describes building an email experimentation pipeline at Shopify after an existing framework for web properties and Shopify admin experiences could not support tests in external channels. The motivating case was a personalized blog-post recommendation algorithm whose incremental benefit needed comparison with generic blog emails, while local randomization and text-file storage created risks around unsubscribed users, shop-email relationships, and overlapping audiences. After requirements gathering, system mapping, mock-table queries, stakeholder review, and technical design review, the author implemented three pipeline phases in PySpark: experiment definitions, consolidation, and filtering with deduplication and exclusion rules. Production output supported ongoing randomization, backfills, category tags, sample limits, and integration with Shopify’s existing analysis framework; the tool was subsequently used across multiple data teams for email experiments.


### [How to Track State with Type 2 Dimensional Models](https://yomu.fyi/post/how-to-track-state-with-type-2-dimensional-models.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 27, 2020
- Type: Tutorial

Application databases often retain only current values in Type 1 dimensions, limiting analysis of historical settings such as feature adoption, retention, and switching behavior. The post explains Type 2 dimensional models through Shopify’s need to track users’ admin languages over time, contrasting application-model changes, scheduled database snapshots, and event logging. Its selected implementation uses Rails after\_commit callbacks to send created or updated records to Kafka, then transforms that event history into records with valid\_from, valid\_to, and is\_current fields using ETL recipes involving PySpark and dbt. The approach provides the required granularity but can miss changes or events, requires delete handling and data-quality checks, and is presented as an iterative solution; MySQL binlogs are described as a more reliable future source.


### [The Journey of Deploying Apache Airflow at Grab](https://yomu.fyi/post/the-journey-of-deploying-apache-airflow-at-grab.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Chandulal Kavar
- Published: Jul 14, 2020
- Type: Problem & solution

Engineering and data teams across Grab originally operated independent Apache Airflow instances, causing duplicate maintenance overhead and frequent job failures around scaling, logging, and dependency management. To resolve this, a dedicated team developed a centralized orchestration platform that runs isolated, containerized Airflow instances per team on Amazon EKS. The platform categorizes deployments into three size tiers and provisions dedicated Redis brokers, RDS metadata stores, and Vault secret sidecars using Terraform and custom Helm charts. Teams customize container images using shared GitLab CI/CD templates, while worker scaling is handled via Kubernetes Horizontal Pod Autoscalers. Today, the platform runs roughly 20 Airflow instances executing between 1,000 and 60,000 daily jobs per instance.


### [Shopify's Data Science & Engineering Foundations](https://yomu.fyi/post/shopify-s-data-science-engineering-foundations.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 18, 2020
- Type: Explainer

Shopify’s Data Science & Engineering team describes the foundations it uses to provide daily insights to internal teams, merchants, and partners across a rapidly changing commerce ecosystem. Its data warehouse follows a shared dimensional-modelling philosophy, with modelled data built on Spark in a single GitHub repository and made broadly queryable through Presto, while ETL jobs are unit tested and prevented from failing silently. Centralized dashboards, reproducible vetted data points, peer review, product-aligned sub-teams, accessible communication, and cross-team collaboration extend those foundations into analysis and decision-making. Together, these practices let team members reuse existing work, produce trusted and understandable metrics, address the specialist gap through collaboration, and derive new metrics with fewer than 50 lines of SQL.


### [Does Southeast Asia Run on Coffee?](https://yomu.fyi/post/does-southeast-asia-run-on-coffee.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Siu Sing Lai
- Published: Mar 26, 2020
- Type: Explainer

GrabFood examined regional coffee ordering trends across major Southeast Asian cities over a nine-month period. Regional coffee orders expanded by 1,400%, with most countries recording their highest order volumes on Wednesdays before tapering off toward the weekend. Singapore and the Philippines deviated from this regional pattern, experiencing spikes in coffee orders on weekends and particularly on Sundays. Daily peak ordering times also differed across markets, peaking at 10:00 AM in Thailand, 2:00 PM in Indonesia, and 4:00 PM in Singapore. In addition to coffee, Green Tea Latte emerged as a top ten beverage item on the platform, accounting for over 25 million delivered cups.


### [GrabChat Much? Talk Data to Me!](https://yomu.fyi/post/grabchat-much-talk-data-to-me.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Jason Lee Jie Shien
- Published: Mar 24, 2020
- Type: Explainer

Grab launched GrabChat in September 2016 to facilitate in-app communication between passengers and driver-partners, subsequently adding features such as instant translation, image sharing, voice audio, and templates. An analysis of messaging data across Singapore, Malaysia, and Indonesia revealed distinct behavioral differences, with Indonesia averaging the highest volume at 5.5 chats per booking. Chat volume and the proportion of location-oriented messages peak between midnight and 4am across all three markets, driven by closed storefronts and obscure pickup spots. Image sharing sees higher utilization in remote areas, while pre-populated templates constitute nearly 50% of driver texts to reduce manual typing while driving. Furthermore, messaging responsiveness, directional guidance, and courtesy terms like saying thank you correlate directly with higher trip completion rates.


### [Data First, SLA Always](https://yomu.fyi/post/data-first-sla-always.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Johan Kok
- Published: Aug 1, 2019
- Type: Problem & solution

Grab's Data Engineering team transitioned from periodic batch ETL ingestion to a real-time change data capture architecture called Trailblazer after dataset sizes exceeded the petabyte mark. The previous system caused severe JDBC timeouts and heavy CPU loads when executing chunked or full-scan queries on unindexed upstream MySQL tables. To solve this, MySQL binary logs are captured via Debezium on Kafka Connect, buffered in Kafka, and ingested into a data lake using Spark Structured Streaming. Checkpoints are decoupled from local storage and persisted in a Redis cluster to simplify ingestion offset overrides and handle ephemeral compute clusters. The system incorporates extensive health monitoring across Airflow, Datadog, and custom services to maintain stream liveliness and avoid Kafka retention breaches.


### [Tourists on GrabChat!](https://yomu.fyi/post/tourists-on-grabchat.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Lara PuReum Yim
- Published: May 22, 2019
- Type: Explainer

Grab examined more than 3.7 million tourist messages across Singapore, Malaysia, and Indonesia sent between December 2018 and March 2019 to evaluate passenger communication patterns. The platform deployed in-house translation and prewritten, auto-translated chat templates to bridge language barriers between international riders and local drivers. Analysis showed that bookings utilizing chat templates experienced a 10% higher ride completion rate than those without. Image-sharing features were most heavily used in high-traffic hubs such as airports, shopping malls, and major tourist centers to aid driver location. Passengers also consistently used messaging to clarify luggage capacity, provide identifiable passenger descriptions, and check pet policies.


### [Bubble Tea Craze on GrabFood!](https://yomu.fyi/post/bubble-tea-craze-on-grabfood.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Lara PuReum Yim
- Published: May 9, 2019
- Type: Explainer

GrabFood recorded a regional average order growth rate of 3,000% for bubble tea across Southeast Asia in 2018. Individual country growth rates ranged from over 250% in Malaysia to more than 8,500% in Indonesia over their respective tracking periods. The customer base for bubble tea expanded by over 12,000%, supported by a 200% increase in merchant outlets to nearly 4,000 locations representing over 1,500 brands. Southeast Asian consumers ordered an average of four cups per person per month, led by Thailand at six cups and the Philippines at five cups. Order timing concentrated primarily around lunchtime meals and midday afternoon breaks.


### [How We Harnessed the Wisdom of Crowds to Improve Restaurant Location Accuracy](https://yomu.fyi/post/how-we-harnessed-the-wisdom-of-crowds-to-improve-restaurant-location-a.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Pravin Kakar
- Published: Apr 2, 2019
- Type: Problem & solution

Grab discovered that abnormally short driver wait times often indicated restaurants registered at incorrect coordinates due to moves or onboarding errors. To fix this, Grab used driver-partner GPS pings, timestamps, and order status updates to infer true food collection locations. The system cleans the data by filtering low-quality GPS pings and isolating the longest temporal streak a driver spends within a predefined radius of the venue. Clusters of inferred pick-up points are then ranked by order volume, the proportion of off-target pick-ups, and median distance errors before routing to mapping operations for verification. This periodic correction workflow achieved a fivefold reduction in order cancellations caused by unfound merchant locations.


[Newer posts](https://yomu.fyi/topic/data-pipelines/page/11.md) · [Older posts](https://yomu.fyi/topic/data-pipelines/page/13.md)