---
title: "The Magic of Merlin: Shopify's New Machine Learning Platform"
description: "Shopify’s machine learning platform team redesigned its infrastructure to accommodate varied use cases, dependencies, data types, integrations, and production requirements. Merlin uses an open-source stack in which on-demand Merlin Workspaces run as short-lived Ray clusters on Kubernetes, while projects isolate code and dependencies in Docker-based environments. Its first iteration focuses on training and batch inference, with users moving from local or Jupyter-based prototypes to automated production workflows through the Merlin API, Airflow, or Oozie. Shopify validated the platform by migrating its product categorization workflows, using Ray Train for distributed TensorFlow training and Ray ActorPool for batch inference; planned extensions include online inference, model lifecycle management, and machine learning monitoring."
---

# The Magic of Merlin: Shopify's New Machine Learning Platform

[Shopify](https://yomu.fyi/company/shopify) · 2023-10-18 · Apr 6, 2022

**Type:** Explainer

## Summary

Shopify’s machine learning platform team redesigned its infrastructure to accommodate varied use cases, dependencies, data types, integrations, and production requirements. Merlin uses an open-source stack in which on-demand Merlin Workspaces run as short-lived Ray clusters on Kubernetes, while projects isolate code and dependencies in Docker-based environments. Its first iteration focuses on training and batch inference, with users moving from local or Jupyter-based prototypes to automated production workflows through the Merlin API, Airflow, or Oozie. Shopify validated the platform by migrating its product categorization workflows, using Ray Train for distributed TensorFlow training and Ray ActorPool for batch inference; planned extensions include online inference, model lifecycle management, and machine learning monitoring.

## Context

Shopify needed a machine learning platform flexible enough to handle different and sometimes conflicting requirements, inputs, data types, dependencies, and integrations while supporting production workflows and enabling data scientists to use suitable tools.

## Approach / What changed

Merlin uses an open-source stack with Merlin Projects for isolated code and dependencies, on-demand Merlin Workspaces implemented as Ray clusters on Kubernetes, and a consolidated API. Ray supports distributed preprocessing, training, and prediction, while users can prototype in Jupyter and automate workflows through Airflow or Oozie.

## Takeaways

- Merlin Workspaces are short-lived Ray clusters deployed on Kubernetes, with environments defined by tasks, dependencies, and required resources.
- Ray Train let Shopify distribute product categorization model training across a Ray cluster while keeping the TensorFlow training logic largely unchanged.
- Product categorization batch inference was distributed with Ray ActorPool; Shopify planned to move to Ray Dataset Pipelines to reduce dependence on dataset partition counts and sizes.

**Tags:** [Docker](https://yomu.fyi/topic/docker), [Kubernetes](https://yomu.fyi/topic/kubernetes), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Python](https://yomu.fyi/topic/python), [Scalability](https://yomu.fyi/topic/scalability)

- Source: [Shopify](https://shopify.engineering/merlin-shopify-machine-learning-platform)
- Source URL: https://shopify.engineering/merlin-shopify-machine-learning-platform
- Ingested by Yomu: 2026-08-30T15:26:58.836Z

[Read original post](https://shopify.engineering/merlin-shopify-machine-learning-platform)