Tangle: An open-source ML experimentation platform built for scale (2025) - Shopify

Summary

Shopify introduces Tangle, an open-source, platform-agnostic machine-learning experimentation platform built for Search and Discovery workflows at commerce scale. It targets six stated failure modes, including repeated data preparation, poor reproducibility, slow deployment, unstructured notebooks, manual query tracking, and limited sharing. Users compose visual directed acyclic graph pipelines from reusable YAML components that wrap containerized or inline command-line programs in multiple languages, while file-based execution moves artifacts through storage. Tangle’s content-based caching reuses results when output content hashes remain identical, including artifacts from still-running executions, rather than relying only on lineage changes. Shopify reports a ten-hour pipeline completing in twenty minutes after one component changes, more than a year of compute time savings, and production use across ranking, semantic search, recommendation, and feature engineering workloads.

Context

Shopify’s Search and Discovery teams faced repeated data preparation, difficult reproduction of old experiments, unstructured notebooks, manual tracking of custom queries, slow deployment, and limited pipeline sharing. The team works with ranking millions of products across billions of queries and reports that 80% of development time is spent on data engineering rather than algorithms.

Approach / What changed

Tangle provides a visual interface for building directed acyclic graph pipelines from reusable YAML components. Components wrap arbitrary containerized or inline command-line programs, allowing different languages and tools to work together without code modifications. The platform records lineage and artifacts, transfers data through storage, and uses global content-based caching to reuse outputs whose content hashes remain identical, including outputs from still-running executions.

Takeaways

• Tangle components are language-neutral YAML specifications that can wrap CLI programs written in Python, Shell, JavaScript, C#, C++, Rust, Java, Go, R, or other CLI-capable languages.
• Content-based caching can reuse downstream results when output content hashes are unchanged; Shopify reports reducing a ten-hour pipeline to twenty minutes after changing one component.
• Tangle records graph structure, logs, artifact metadata, and metrics for each run, allowing team members to clone, modify, and resubmit experiments with tracked lineage.

Related reading