How We’re Solving Data Discovery Challenges at Shopify

Summary

Shopify describes data discovery as a response to growing numbers of data assets and the difficulty of finding, governing, and using them across the data lifecycle. At Shopify, pre-Artifact discovery relied on people, Slack, GitHub, and job logs, while teams lacked granular ownership, change-impact visibility, and documented reporting context. They built Artifact, a search-and-browse tool backed by a centralized metadata model, ingestion jobs, Elasticsearch indexes, GraphQL APIs, and a graph database for filterable upstream and downstream lineage. Its landing page supports browsing and plain-English search, while details pages expose documentation, schema, descriptions, usage, ownership, and lineage; initial scope prioritized frequently used, high-impact, queryable assets and downstream reports and dashboards. Since launch in early 2020, weekly Data team usage exceeded 30% and monthly retention exceeded 50%, above expectations, with future work planned for additional assets, integrations, stewardship, change management, notifications, and metadata APIs.

Context

Shopify faced three data discovery and management challenges: curation, governance, and accessibility. Teams struggled to find existing assets, understand ownership and downstream change impact, and surface documented insights from reports and dashboards. The existing process depended on in-person requests, Slack, GitHub, job logs, and other fragmented sources.

Approach / What changed

Shopify built Artifact as a search-and-browse tool on a centralized data model that aggregates metadata from multiple data processes. Its ingestion pipeline pulls metadata, builds a dependency graph for lineage, and stores information in Elasticsearch indexes exposed through GraphQL APIs. The initial release prioritized frequently used, high-impact queryable assets and their downstream reports and dashboards.

Takeaways

• Before Artifact, 80% of Shopify’s Data team said discovery hindered their ability to deliver results; after release, that figure fell to 41%.
• Artifact’s lineage feature uses a graph database and lets users search and filter dependencies by source, upstream or downstream direction, and direct or indirect distance.
• Shopify chose metadata-pulling jobs over provider APIs to improve speed to market, while treating metadata APIs for each data service as a future optimization.

Related reading