Shopify's Data Science & Engineering Foundations

Summary

Shopify’s Data Science & Engineering team describes the foundations it uses to provide daily insights to internal teams, merchants, and partners across a rapidly changing commerce ecosystem. Its data warehouse follows a shared dimensional-modelling philosophy, with modelled data built on Spark in a single GitHub repository and made broadly queryable through Presto, while ETL jobs are unit tested and prevented from failing silently. Centralized dashboards, reproducible vetted data points, peer review, product-aligned sub-teams, accessible communication, and cross-team collaboration extend those foundations into analysis and decision-making. Together, these practices let team members reuse existing work, produce trusted and understandable metrics, address the specialist gap through collaboration, and derive new metrics with fewer than 50 lines of SQL.

Context

Shopify’s Data Science & Engineering team supports internal teams, merchants, and partners with daily insights in a rapidly changing commerce environment. The organization emphasizes shared foundations so data can be discovered, trusted, reused, and connected to decisions across its commerce ecosystem.

Approach / What changed

The team uses shared dimensional modelling, a Spark-based modelling platform in one GitHub repository, company-wide Presto access, tested and observable ETL, centralized dashboards, reproducible vetted data points, peer review, product-aligned sub-teams, clear communication, cross-team task forces, and leadership engagement with data-informed recommendations.

Takeaways

• Shopify’s shared dimensional-modelling philosophy standardizes fact and dimension tables and unresolved rows, so analysts can join dimensions without sneaky nulls silently destroying rows.
• All data pipeline jobs are unit tested for errors and edge cases, and failures generate alerts rather than being allowed to fail silently.
• Vetted data points store the original question, answer, and generating code together, with the goal that a result remains unchanged over time.

Related reading