Unlocking Real-time Predictions with Shopify's Machine Learning Platform

Summary

Shopify’s Merlin machine learning platform was expanded to support online inference, allowing trained models to return predictions as input arrives rather than through periodic batch jobs. The design targets varied Shopify use cases and requirements, including support for TensorFlow, PyTorch and XGBoost, low latency, autoscaling, rolling deployments, observability and model updates. Each use case runs as a dedicated service on Google Kubernetes Engine, with a Python serving layer that loads models and artifacts, exposes an inference endpoint, and can use the Comet ML model registry and Pano feature store. Separate staging and production configurations let teams tune resources and test model versions before release; Shopify reports that online inference is providing the intended low latency, scalability and faster iteration, while ensemble inference, platform-based monitoring and continuous training remain planned areas.

Context

Shopify needed real-time predictions for user-facing applications and services, while supporting machine learning teams with different use cases, requirements, models and dependencies. The platform had to provide low latency at Shopify scale while balancing cost, use-case requirements and the teams’ ability to operate machine learning services in production.

Approach / What changed

Merlin Online Inference deploys each online-inference use case as a dedicated service on Shopify’s Google Kubernetes Engine clusters. A Python serving layer loads models and artifacts, exposes inference endpoints and supports reusable workflow logic, while service configuration controls resources, autoscaling and environment-specific settings. The services integrate with Comet ML for model artifacts and Pano for low-latency feature access, and use Shopify’s internal deployment ecosystem.

Takeaways

• Each Merlin online-inference use case runs in its own Kubernetes namespace and can be configured to autoscale using service-specific parameters and metrics.
• The serving layer supports model endpoints implemented in Python and is designed to work with different machine learning libraries, including TensorFlow, PyTorch and XGBoost.
• Merlin’s planned areas include ensemble models or inference graphs, platform-based monitoring for drift, and more automated continuous training and model deployment.

Related reading