Introducing Fly Kubernetes

Summary

Fly.io introduces Fly Kubernetes (FKS), an implementation of Kubernetes that runs on its globally distributed container platform for users who want Kubernetes tooling and declarative orchestration. Fly.io says its earlier Nomad-based approach struggled with global scheduling at more than 200,000 applications and with placing workloads in users’ requested regions, so it moved state to individual workers and exposed the Fly Machines API, whose placement requests fail quickly rather than retry elsewhere. FKS combines K3s with Virtual Kubelet, mapping pod lifecycle operations to Fly Machines API calls while running K3s and Virtual Kubelet on one Fly Machine. A generated kubeconfig enables kubectl access, and Pods run on separate Fly Machines as the platform scales the cluster. The design provides declarative replica, autoscaling, and volume definitions plus automatic recovery for failed Machines, while retaining Fly Machines’ trade-off of failing when requested placement is unavailable.

Takeaways

1. 1

   The Fly Machines API keeps application state local to workers and uses a market-style scheduler that bids on regional resources instead of maintaining a globally consistent scheduler state.

2. 2

   FKS maps Virtual Kubelet pod lifecycle operations, including CreatePod and UpdatePod, to Fly Machines API endpoints for creating and updating Machines.

3. 3

   Kubernetes deployments declare replica counts, autoscaling rules, and volumes, while Kubernetes automatically works to restore Machines to the declared state when they go down.