Loading post…
Fly.ioProblem & solution
Making Machines Move
Summary
Fly.io needed to restore the ability to drain physical workers after attached NVMe volumes made stateful Fly Machines tied to particular servers. Snapshot restoration could lose data, while killing a machine before copying its volume caused unacceptable downtime; copying first risked losing writes. The solution was an asynchronous clone operation built around Linux device mapper’s dm-clone: a new volume is created immediately, reads of unhydrated blocks are fetched from the original over iSCSI, writes go to the destination, and background copying rehydrates the rest. The work also required orchestration across flyd and a fix for Fly Postgres configurations containing literal 6PN IPv6 addresses. After cautious early migrations, the infrastructure team could drain hosts without much ceremony by summer 2024, though routine automated rebalancing remained a future goal.
Context
Attached NVMe volumes kept application data on particular worker servers, making the existing stateless-worker drain process unsuitable for stateful Fly Machines. Restoring snapshots could lose data, while copying volumes during migration introduced either data loss or unacceptable downtime.
Approach / What changed
Fly.io introduced asynchronous volume cloning using Linux device mapper’s dm-clone. Cloned volumes could be attached to new machines immediately; reads of blocks not yet copied were hydrated from the original volume over iSCSI, while writes went to the destination and background copying completed the transfer. The work also included orchestration changes and coordinated fixes for literal 6PN IPv6 addresses in Fly Postgres configurations.
Takeaways
dm-clone tracks hydrated and unhydrated regions with metadata, routing reads for missing blocks to the source while background kcopyd rehydrates the clone.
The migration sequence changed from slow kill, copy, boot to fast kill, clone, boot, allowing migration speed to approach stateless migration.
By summer 2024, Fly.io’s infrastructure team could drain hosts without much ceremony, but fully automated workload rebalancing was still a future goal.
Source and freshness
- Source
- fly.io
- Published
- Added to Yomu
- Last updated
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.