How to Build an Experiment Pipeline from Scratch

Summary

The post describes building an email experimentation pipeline at Shopify after an existing framework for web properties and Shopify admin experiences could not support tests in external channels. The motivating case was a personalized blog-post recommendation algorithm whose incremental benefit needed comparison with generic blog emails, while local randomization and text-file storage created risks around unsubscribed users, shop-email relationships, and overlapping audiences. After requirements gathering, system mapping, mock-table queries, stakeholder review, and technical design review, the author implemented three pipeline phases in PySpark: experiment definitions, consolidation, and filtering with deduplication and exclusion rules. Production output supported ongoing randomization, backfills, category tags, sample limits, and integration with Shopify’s existing analysis framework; the tool was subsequently used across multiple data teams for email experiments.

Context

Shopify’s existing experimentation framework could randomize shops and pre-login web sessions, but it did not support email experiments because assignment was not triggered by a site visit. Data scientists were using ad hoc randomization and local text files, creating discoverability, unsubscribe, shop-email relationship, cross-contamination, and audience-overlap problems.

Approach / What changed

The pipeline uses three phases implemented in PySpark: experiment definition files specify eligibility, variations, regions, categories, and sample limits; a many-to-one transform consolidates experiments and appends ongoing randomizations; and filtering applies experiment-priority, multi-shop exclusion, deduplication, and feature-enrichment rules. The resulting table integrates with the existing analysis framework.

Takeaways

• Subjects in a current experiment are excluded from other experiments for at least 30 days, with an override available for longer periods when testing higher-risk variations.
• The output is one row per email, shop, and experiment, with timing and experiment attributes designed to join email-platform engagement data and existing analysis tools.
• Small, incremental PySpark pull requests made the several-thousand-line implementation easier for reviewers to assess and production failures easier to debug.

Related reading