Shopify's Playbook for Scaling Machine Learning

Summary

Shopify describes how its machine-learning practice grew from a first order-fraud product into a portfolio used across products such as Shopify Capital, product categorization, and Help Center search. The playbook starts with choosing a user-important problem, validating trustworthy and accessible data, tracing downstream dependencies, understanding existing solutions, and optimizing for product outcomes rather than model scores alone. To productionize models, Shopify recommends well-defined training, verification, and testing pipelines, deployment decisions based on volume and user commitment, and platform practices that encode validated standards without abstracting too early. Its fraud pipeline builds models in Python, serializes them with PMML for deployment in Ruby-based production systems, and uses Apache Airflow to schedule operations. The account concludes that scaling is a full-stack problem requiring data scientists and data engineers to align on one roadmap and goal.

Context

Shopify needed a pragmatic way to decide where to begin with machine learning amid widespread noise about best practices. The first problem had to matter to users and the business, have trustworthy and accessible data, account for downstream workflows, and improve on an existing solution. For order fraud, the existing rule-based system produced many false positives, forcing merchants to investigate or cancel legitimate orders.

Approach / What changed

The playbook combines domain-embedded data scientists with structured problem selection, measurable product outcomes, and data-engineering foundations. Shopify recommends defined pipelines for training, verification, and testing; deployment choices based on model volume and user commitment; and platform features that encode proven practices such as mandatory backtesting. Automation handles scheduling, checks, versioning, and deployment operations, while Python models are serialized with PMML for Ruby production systems and run through Apache Airflow.

Takeaways

• Shopify optimized order-fraud detection for the highest number of valid sales rather than for zero fraud or isolated metrics such as accuracy, precision, and recall.
• The production pipeline uses Python for model building, PMML to make models language-independent, Ruby-based production services, and Apache Airflow scheduling.
• Shopify advises encoding best practices into a platform only after testing them across multiple models, while keeping humans involved to investigate flagged anomalies.

Related reading