Loading post…
Shopify2023-10-18Explainer
SkyPilot at Shopify: Multi-cloud GPUs without the pain
Summary
Shopify uses SkyPilot to run machine-learning training workloads across GPU clusters on Nebius and GCP, addressing scarce hardware, fragmented capacity, and provider-specific interfaces. Persistent Kubernetes clusters remain managed by Shopify, while SkyPilot launches jobs and a company plugin routes requests, validates labels, injects configuration, and applies organizational policies. H200 workloads go to Nebius for InfiniBand-equipped distributed training, while L4 and CPU-only jobs go to GCP; labels also connect jobs to cost ownership, Kueue quotas, priority classes, and development-environment rules. The plugin automatically configures InfiniBand access, shared caches, and temporary storage, reducing manual setup and speeding repeated workloads. The resulting YAML-based interface preserves direct resource control while hiding cloud-selection decisions, giving teams common workflows as providers, prices, and GPU availability change.
Context
Shopify's machine-learning workloads require many GPUs, but GPUs are scarce and fragmented across clouds with different APIs and configurations. The organization also needed multi-team management, cost tracking, fair scheduling, and a way to run jobs on existing clusters without requiring engineers to learn each cloud's setup.
Approach / What changed
Shopify runs SkyPilot as a launcher for persistent Kubernetes clusters on Nebius and GCP. A custom plugin connected to SkyPilot's policy engine intercepts requests, routes them by accelerator or provider override, validates labels, injects Nebius-specific pod settings and shared caches, and connects jobs to cost ownership, Kueue quotas, priority classes, development-environment behavior, and storage cleanup policies.
Takeaways
SkyPilot routes H200 workloads to Nebius and L4 or CPU-only workloads to GCP, while honoring an explicit force_provider_selection setting.
Shopify's plugin automatically adds Nebius InfiniBand requirements, including /dev/infiniband, IPC_LOCK, and libibverbs1, and mounts shared caches for Python packages and model weights.
Job labels enforce cost ownership and connect workloads to Kueue fair-share scheduling and priority-based preemption; development environments receive interactive priority and are limited to one GPU.
Source and freshness
- Source
- shopify.engineering
- Published
- Added to Yomu
- Last updated
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.