How to Run Isolated Tenant Kubernetes Clusters on Shared GPU Infrastructure

Summary

Organizations sharing a single Kubernetes cluster across multiple teams frequently encounter conflicting custom resource definitions, overlapping role-based access control, and difficulties budgeting GPU capacity. To address these operational conflicts without provisioning separate hardware, administrators can deploy vCluster alongside KAI Scheduler on a shared host cluster. vCluster supplies each team with a dedicated, virtualized control plane featuring its own API server and permissions, while KAI Scheduler dynamically time-slices GPU resources through hierarchical queues. In a demonstration using an NVIDIA L40S GPU on MicroK8s, three separate tenant clusters successfully scheduled workloads with fractional allocations of 0.33 GPU each. The combination achieves tenant autonomy and visibility isolation while consolidating AI workloads on shared physical infrastructure.

Takeaways

1. 1

   KAI Scheduler handles pod scheduling and fractional GPU allocation via queue hierarchies, but it does not enforce hardware-level GPU memory isolation under software time-slicing.

2. 2

   Pods target KAI Scheduler by specifying kai-scheduler as their schedulerName, alongside queue labels and gpu-fraction annotations, allowing kube-scheduler to handle standard workloads.

3. 3

   vCluster isolates API servers, custom resource definitions, namespaces, and access controls for each team while sharing underlying worker nodes and GPU resources.