---
title: "JIT WireGuard"
description: "Fly.io describes how its gateways provisioned WireGuard peers for flyctl and why the old design became unreliable and slow. NATS delivery losses caused failed provisioning, while CI-generated peers accumulated because gateways never removed them, leaving hundreds of thousands of stale kernel entries that slowed reloads and contributed to kernel panics. The replacement detects handshake-initiation packets with BPF or a WebSockets packet hook, runs enough Noise cryptography to recover the initiator’s public key, then fetches matching peer data over an internal HTTP API and installs it on demand. A rate-limited SQLite cache supports cleanup jobs, and the gateway can use the observed four-tuple to initiate back toward flyctl and accelerate handshakes. After weeks in production, stale peers rounded to none, with faster setup and rebooting that no longer required reloading unused entries."
---

# JIT WireGuard

[Fly.io](https://yomu.fyi/company/fly-io) · Mar 12, 2024

**Type:** Problem & solution

## Summary

Fly.io describes how its gateways provisioned WireGuard peers for flyctl and why the old design became unreliable and slow. NATS delivery losses caused failed provisioning, while CI-generated peers accumulated because gateways never removed them, leaving hundreds of thousands of stale kernel entries that slowed reloads and contributed to kernel panics. The replacement detects handshake-initiation packets with BPF or a WebSockets packet hook, runs enough Noise cryptography to recover the initiator’s public key, then fetches matching peer data over an internal HTTP API and installs it on demand. A rate-limited SQLite cache supports cleanup jobs, and the gateway can use the observed four-tuple to initiate back toward flyctl and accelerate handshakes. After weeks in production, stale peers rounded to none, with faster setup and rebooting that no longer required reloading unused entries.

## Context

Fly.io’s gateways accumulated hundreds of thousands of WireGuard peers created by flyctl, especially from CI jobs that could not reconnect to existing peers. NATS delivery losses also made peer provisioning unreliable, while stale kernel state slowed WireGuard operations, gateway reboots, and sometimes contributed to kernel panics.

## Approach / What changed

The gateway detects incoming WireGuard handshake initiations with a BPF filter or a packet hook in its WebSockets transport, performs the initial Noise cryptography to identify the peer, and fetches its configuration through an internal HTTP API. A rate-limited SQLite cache and cleanup jobs allow stale peers to be removed, while the observed connection tuple enables the gateway to initiate a response immediately.

## Takeaways

- The incoming WireGuard handshake-initiation packet can be identified with the filter udp and dst port 51820 and udp\[8\] = 1.
- Identifying a peer requires running enough of the Noise handshake to decrypt its identity; the implementation is described as roughly 200 lines of code.
- In production, the JIT design reduced gateway stale peers from thousands or hundreds of thousands to a count that rounds to none, while reducing reboot reload work.

**Tags:** [BPF](https://yomu.fyi/topic/bpf), [SQLite](https://yomu.fyi/topic/sqlite), [WireGuard](https://yomu.fyi/topic/wireguard)

- Source: [Fly.io](https://fly.io/blog/jit-wireguard-peers)
- Source URL: https://fly.io/blog/jit-wireguard-peers
- Ingested by Yomu: 2026-09-03T01:25:07.034Z

[Read original post](https://fly.io/blog/jit-wireguard-peers)