Corrosion

Summary

Corrosion is Fly.io’s open-source service discovery and state synchronization system, built to distribute routing data across globally deployed edge proxies and worker servers. It replaces centralized consensus-oriented designs with a Rust program that propagates a SQLite database through SWIM-based gossip and QUIC, using cr-sqlite’s CRDT tables and logical timestamps to reconcile updates without locking, a central server, or distributed consensus. The system’s design follows link-state routing ideas and exploits Fly.io’s fully connected WireGuard mesh, while worker-owned state keeps conflicts uncommon. The post also documents failures, including a contagious Rust RWLock deadlock, a nullable-column migration that triggered large-scale reconciliation, and the need for backups, whole-dataset republishing, testing, and regionalization to reduce blast radius. Corrosion now converges in seconds across thousands of workers.

Takeaways

1. 1

   Corrosion combines SWIM membership gossip with QUIC update distribution and cr-sqlite CRDT tables, which apply changes using last-write-wins logical timestamps.

2. 2

   Worker-owned state avoids bottlenecking on a globally consistent central scheduler because updates from different workers almost never conflict.

3. 3

   Regionalization separates fine-grained regional clusters from a global application-to-region map, reducing the blast radius of state-related bugs; checkpoint backups support rebuilding the database.