---
title: "Making Machines Move"
description: "Fly.io needed to restore the ability to drain physical workers after attached NVMe volumes made stateful Fly Machines tied to particular servers. Snapshot restoration could lose data, while killing a machine before copying its volume caused unacceptable downtime; copying first risked losing writes. The solution was an asynchronous clone operation built around Linux device mapper’s dm-clone: a new volume is created immediately, reads of unhydrated blocks are fetched from the original over iSCSI, writes go to the destination, and background copying rehydrates the rest. The work also required orchestration across flyd and a fix for Fly Postgres configurations containing literal 6PN IPv6 addresses. After cautious early migrations, the infrastructure team could drain hosts without much ceremony by summer 2024, though routine automated rebalancing remained a future goal."
---

# Making Machines Move

[Fly.io](https://yomu.fyi/company/fly-io) · Jul 30, 2024

**Type:** Problem & solution

## Summary

Fly.io needed to restore the ability to drain physical workers after attached NVMe volumes made stateful Fly Machines tied to particular servers. Snapshot restoration could lose data, while killing a machine before copying its volume caused unacceptable downtime; copying first risked losing writes. The solution was an asynchronous clone operation built around Linux device mapper’s dm-clone: a new volume is created immediately, reads of unhydrated blocks are fetched from the original over iSCSI, writes go to the destination, and background copying rehydrates the rest. The work also required orchestration across flyd and a fix for Fly Postgres configurations containing literal 6PN IPv6 addresses. After cautious early migrations, the infrastructure team could drain hosts without much ceremony by summer 2024, though routine automated rebalancing remained a future goal.

## Context

Attached NVMe volumes kept application data on particular worker servers, making the existing stateless-worker drain process unsuitable for stateful Fly Machines. Restoring snapshots could lose data, while copying volumes during migration introduced either data loss or unacceptable downtime.

## Approach / What changed

Fly.io introduced asynchronous volume cloning using Linux device mapper’s dm-clone. Cloned volumes could be attached to new machines immediately; reads of blocks not yet copied were hydrated from the original volume over iSCSI, while writes went to the destination and background copying completed the transfer. The work also included orchestration changes and coordinated fixes for literal 6PN IPv6 addresses in Fly Postgres configurations.

## Takeaways

- dm-clone tracks hydrated and unhydrated regions with metadata, routing reads for missing blocks to the source while background kcopyd rehydrates the clone.
- The migration sequence changed from slow kill, copy, boot to fast kill, clone, boot, allowing migration speed to approach stateless migration.
- By summer 2024, Fly.io’s infrastructure team could drain hosts without much ceremony, but fully automated workload rebalancing was still a future goal.

**Tags:** [Distributed Systems](https://yomu.fyi/topic/distributed-systems), [Fly Machines](https://yomu.fyi/topic/fly-machines), [iSCSI](https://yomu.fyi/topic/iscsi), [Linux](https://yomu.fyi/topic/linux)

- Source: [Fly.io](https://fly.io/blog/machine-migrations)
- Source URL: https://fly.io/blog/machine-migrations
- Ingested by Yomu: 2026-09-02T16:00:48.726Z

[Read original post](https://fly.io/blog/machine-migrations)