Unlocking Real-time Predictions with Shopify's Machine Learning Platform

Summary

Shopify’s Merlin machine learning platform was expanded to support online inference, allowing trained models to return predictions as input arrives rather than through periodic batch jobs. The design targets varied Shopify use cases and requirements, including support for TensorFlow, PyTorch and XGBoost, low latency, autoscaling, rolling deployments, observability and model updates. Each use case runs as a dedicated service on Google Kubernetes Engine, with a Python serving layer that loads models and artifacts, exposes an inference endpoint, and can use the Comet ML model registry and Pano feature store. Separate staging and production configurations let teams tune resources and test model versions before release; Shopify reports that online inference is providing the intended low latency, scalability and faster iteration, while ensemble inference, platform-based monitoring and continuous training remain planned areas.

Takeaways

1. 1

   Each Merlin online-inference use case runs in its own Kubernetes namespace and can be configured to autoscale using service-specific parameters and metrics.

2. 2

   The serving layer supports model endpoints implemented in Python and is designed to work with different machine learning libraries, including TensorFlow, PyTorch and XGBoost.

3. 3

   Merlin’s planned areas include ensemble models or inference graphs, platform-based monitoring for drift, and more automated continuous training and model deployment.