---
title: "SkyPilot at Shopify: Multi-cloud GPUs without the pain"
description: "Shopify uses SkyPilot to run machine-learning training workloads across GPU clusters on Nebius and GCP, addressing scarce hardware, fragmented capacity, and provider-specific interfaces. Persistent Kubernetes clusters remain managed by Shopify, while SkyPilot launches jobs and a company plugin routes requests, validates labels, injects configuration, and applies organizational policies. H200 workloads go to Nebius for InfiniBand-equipped distributed training, while L4 and CPU-only jobs go to GCP; labels also connect jobs to cost ownership, Kueue quotas, priority classes, and development-environment rules. The plugin automatically configures InfiniBand access, shared caches, and temporary storage, reducing manual setup and speeding repeated workloads. The resulting YAML-based interface preserves direct resource control while hiding cloud-selection decisions, giving teams common workflows as providers, prices, and GPU availability change."
---

# SkyPilot at Shopify: Multi-cloud GPUs without the pain

[Shopify](https://yomu.fyi/company/shopify) · 2023-10-18 · Jan 26, 2026

**Type:** Explainer

## Summary

Shopify uses SkyPilot to run machine-learning training workloads across GPU clusters on Nebius and GCP, addressing scarce hardware, fragmented capacity, and provider-specific interfaces. Persistent Kubernetes clusters remain managed by Shopify, while SkyPilot launches jobs and a company plugin routes requests, validates labels, injects configuration, and applies organizational policies. H200 workloads go to Nebius for InfiniBand-equipped distributed training, while L4 and CPU-only jobs go to GCP; labels also connect jobs to cost ownership, Kueue quotas, priority classes, and development-environment rules. The plugin automatically configures InfiniBand access, shared caches, and temporary storage, reducing manual setup and speeding repeated workloads. The resulting YAML-based interface preserves direct resource control while hiding cloud-selection decisions, giving teams common workflows as providers, prices, and GPU availability change.

## Context

Shopify's machine-learning workloads require many GPUs, but GPUs are scarce and fragmented across clouds with different APIs and configurations. The organization also needed multi-team management, cost tracking, fair scheduling, and a way to run jobs on existing clusters without requiring engineers to learn each cloud's setup.

## Approach / What changed

Shopify runs SkyPilot as a launcher for persistent Kubernetes clusters on Nebius and GCP. A custom plugin connected to SkyPilot's policy engine intercepts requests, routes them by accelerator or provider override, validates labels, injects Nebius-specific pod settings and shared caches, and connects jobs to cost ownership, Kueue quotas, priority classes, development-environment behavior, and storage cleanup policies.

## Takeaways

- SkyPilot routes H200 workloads to Nebius and L4 or CPU-only workloads to GCP, while honoring an explicit force\_provider\_selection setting.
- Shopify's plugin automatically adds Nebius InfiniBand requirements, including /dev/infiniband, IPC\_LOCK, and libibverbs1, and mounts shared caches for Python packages and model weights.
- Job labels enforce cost ownership and connect workloads to Kueue fair-share scheduling and priority-based preemption; development environments receive interactive priority and are limited to one GPU.

**Tags:** [Google Cloud](https://yomu.fyi/topic/gcp), [Kubernetes](https://yomu.fyi/topic/kubernetes), [Open Source](https://yomu.fyi/topic/open-source), [Performance](https://yomu.fyi/topic/performance), [Scalability](https://yomu.fyi/topic/scalability)

- Source: [Shopify](https://shopify.engineering/skypilot)
- Source URL: https://shopify.engineering/skypilot
- Ingested by Yomu: 2026-08-30T13:18:28.595Z

[Read original post](https://shopify.engineering/skypilot)