SkyPilot at Shopify: Multi-cloud GPUs without the pain

Summary

Shopify uses SkyPilot to run machine-learning training workloads across GPU clusters on Nebius and GCP, addressing scarce hardware, fragmented capacity, and provider-specific interfaces. Persistent Kubernetes clusters remain managed by Shopify, while SkyPilot launches jobs and a company plugin routes requests, validates labels, injects configuration, and applies organizational policies. H200 workloads go to Nebius for InfiniBand-equipped distributed training, while L4 and CPU-only jobs go to GCP; labels also connect jobs to cost ownership, Kueue quotas, priority classes, and development-environment rules. The plugin automatically configures InfiniBand access, shared caches, and temporary storage, reducing manual setup and speeding repeated workloads. The resulting YAML-based interface preserves direct resource control while hiding cloud-selection decisions, giving teams common workflows as providers, prices, and GPU availability change.

Context

Shopify's machine-learning workloads require many GPUs, but GPUs are scarce and fragmented across clouds with different APIs and configurations. The organization also needed multi-team management, cost tracking, fair scheduling, and a way to run jobs on existing clusters without requiring engineers to learn each cloud's setup.

Approach / What changed

Shopify runs SkyPilot as a launcher for persistent Kubernetes clusters on Nebius and GCP. A custom plugin connected to SkyPilot's policy engine intercepts requests, routes them by accelerator or provider override, validates labels, injects Nebius-specific pod settings and shared caches, and connects jobs to cost ownership, Kueue quotas, priority classes, development-environment behavior, and storage cleanup policies.

Takeaways

• SkyPilot routes H200 workloads to Nebius and L4 or CPU-only workloads to GCP, while honoring an explicit force_provider_selection setting.
• Shopify's plugin automatically adds Nebius InfiniBand requirements, including /dev/infiniband, IPC_LOCK, and libibverbs1, and mounts shared caches for Python packages and model weights.
• Job labels enforce cost ownership and connect workloads to Kueue fair-share scheduling and priority-based preemption; development environments receive interactive priority and are limited to one GPU.

Related reading