Docker without Docker

Summary

Fly.io explains how it accepts OCI container images while avoiding Docker at runtime, choosing Firecracker micro-VMs because Docker’s isolation is not strong enough for its high-density multitenant environment. It describes OCI images as ordered tarball layers, showing how registry authentication, manifests, architecture selection, blob downloads, and configuration data are enough to reconstruct a container filesystem and entrypoint. On the platform side, flyctl uses Docker’s client library to push images, while a custom Go repository server adds token-based authorization, repository rewriting, and handling for cross-repository blob mounts and registry state. Servers use containerd as a cache over an LVM2 thin pool, leases and copy-on-write snapshots to provision VM filesystems quickly, and a Rust init configures mounts, networking, DNS, SSH, and the entrypoint before Firecracker runs the workload.

Takeaways

1. 1

   OCI’s tar-based layers are simple to pull and unpack, but tar has weak standardization, poor random access and incremental-update behavior, and can duplicate large files after small changes.

2. 2

   The custom repository server wraps Docker registry handlers with authorization middleware; it must rewrite cross-repository blob mounts and _state URLs, sharing an HMAC key with the registry.

3. 3

   containerd uses leases for snapshot garbage collection, while LVM2 copy-on-write lets multiple VM instances share image blocks without stepping on one another. Subsequent deployments are faster than the platform’s logging.