How We’re Solving Data Discovery Challenges at Shopify

Summary

Shopify describes data discovery as a response to growing numbers of data assets and the difficulty of finding, governing, and using them across the data lifecycle. At Shopify, pre-Artifact discovery relied on people, Slack, GitHub, and job logs, while teams lacked granular ownership, change-impact visibility, and documented reporting context. They built Artifact, a search-and-browse tool backed by a centralized metadata model, ingestion jobs, Elasticsearch indexes, GraphQL APIs, and a graph database for filterable upstream and downstream lineage. Its landing page supports browsing and plain-English search, while details pages expose documentation, schema, descriptions, usage, ownership, and lineage; initial scope prioritized frequently used, high-impact, queryable assets and downstream reports and dashboards. Since launch in early 2020, weekly Data team usage exceeded 30% and monthly retention exceeded 50%, above expectations, with future work planned for additional assets, integrations, stewardship, change management, notifications, and metadata APIs.

Takeaways

1. 1

   Before Artifact, 80% of Shopify’s Data team said discovery hindered their ability to deliver results; after release, that figure fell to 41%.

2. 2

   Artifact’s lineage feature uses a graph database and lets users search and filter dependencies by source, upstream or downstream direction, and direct or indirect distance.

3. 3

   Shopify chose metadata-pulling jobs over provider APIs to improve speed to market, while treating metadata APIs for each data service as a future optimization.