Fly.ioProblem & solution
Fly.io describes how its gateways provisioned WireGuard peers for flyctl and why the old design became unreliable and slow. NATS delivery losses caused failed provisioning, while CI-generated peers accumulated because gateways never removed them, leaving hundreds of thousands of stale kernel entries that slowed reloads and contributed to kernel panics. The replacement detects handshake-initiation packets with BPF or a WebSockets packet hook, runs enough Noise cryptography to recover the initiator’s public key, then fetches matching peer data over an internal HTTP API and installs it on demand. A rate-limited SQLite cache supports cleanup jobs, and the gateway can use the observed four-tuple to initiate back toward flyctl and accelerate handshakes. After weeks in production, stale peers rounded to none, with faster setup and rebooting that no longer required reloading unused entries.
Fly.io’s gateways accumulated hundreds of thousands of WireGuard peers created by flyctl, especially from CI jobs that could not reconnect to existing peers. NATS delivery losses also made peer provisioning unreliable, while stale kernel state slowed WireGuard operations, gateway reboots, and sometimes contributed to kernel panics.
The gateway detects incoming WireGuard handshake initiations with a BPF filter or a packet hook in its WebSockets transport, performs the initial Noise cryptography to identify the peer, and fetches its configuration through an internal HTTP API. A rate-limited SQLite cache and cleanup jobs allow stale peers to be removed, while the observed connection tuple enables the gateway to initiate a response immediately.
The incoming WireGuard handshake-initiation packet can be identified with the filter udp and dst port 51820 and udp[8] = 1.
Identifying a peer requires running enough of the Noise handshake to decrypt its identity; the implementation is described as roughly 200 lines of code.
In production, the JIT design reduced gateway stale peers from thousands or hundreds of thousands to a count that rounds to none, while reducing reboot reload work.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
BPF is not one hook but a set of attachment points, and each one sees different traffic. This diagram maps XDP, TC, socket-level BPF LSM, and sched_ext so you can pick the layer that can actually enforce your rule.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
