Loading…
Topic
WireGuard
17 posts about WireGuard. Every summary links to the original.
- 17 posts
- 2 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Your sandbox can now serve requests
Sandboxes have traditionally been outbound-only machines, unable to receive inbound connections because they sit behind NAT with no public IP or open ports. BoxCompute now enables sandboxes to serve requests by using Tailcat, an open-source library from Tailscale that performs NAT traversal. Tailcat establishes a direct peer-to-peer WireGuard-encrypted path between the client and sandbox, using a DERP relay only for the initial handshake. This approach avoids the latency and bandwidth bottlenecks of relay-based tools like ngrok or Cloudflare Tunnel, and keeps the relay out of the data path. The feature is rolling out as an early preview with scoped, short-lived grants for selected TCP ports.
Corrosion
Corrosion is Fly.io’s open-source service discovery and state synchronization system, built to distribute routing data across globally deployed edge proxies and worker servers. It replaces centralized consensus-oriented designs with a Rust program that propagates a SQLite database through SWIM-based gossip and QUIC, using cr-sqlite’s CRDT tables and logical timestamps to reconcile updates without locking, a central server, or distributed consensus. The system’s design follows link-state routing ideas and exploits Fly.io’s fully connected WireGuard mesh, while worker-owned state keeps conflicts uncommon. The post also documents failures, including a contagious Rust RWLock deadlock, a nullable-column migration that triggered large-scale reconciliation, and the need for backups, whole-dataset republishing, testing, and regionalization to reduce blast radius. Corrosion now converges in seconds across thousands of workers.
JIT WireGuard
Fly.io describes how its gateways provisioned WireGuard peers for flyctl and why the old design became unreliable and slow. NATS delivery losses caused failed provisioning, while CI-generated peers accumulated because gateways never removed them, leaving hundreds of thousands of stale kernel entries that slowed reloads and contributed to kernel panics. The replacement detects handshake-initiation packets with BPF or a WebSockets packet hook, runs enough Noise cryptography to recover the initiator’s public key, then fetches matching peer data over an internal HTTP API and installs it on demand. A rate-limited SQLite cache supports cleanup jobs, and the gateway can use the observed four-tuple to initiate back toward flyctl and accelerate handshakes. After weeks in production, stale peers rounded to none, with faster setup and rebooting that no longer required reloading unused entries.
Fly Kubernetes does more now
Fly Kubernetes is moving from early access to a closed beta as a managed Kubernetes service backed by Fly.io infrastructure. It provides the Kubernetes API while mapping core primitives to Fly resources: containers become Firecracker microVMs through flyd and Firecracker, pods become Fly Machines, networking uses an internal WireGuard mesh, services use the Fly Proxy, and storage uses Fly Volumes. Virtual Kubelet, running alongside K3s, receives pod requests and deploys them through the Fly Machines API, removing user-managed node capacity planning and hardware management. The beta supports workloads, including GPUs, across available regions, with Deployments/ReplicaSets, Services, DNS, and ephemeral and persistent volumes, but only part of the Kubernetes API. Missing capabilities include multi-container pods, StatefulSets, network policies, horizontal pod autoscaling, and emptyDir volumes; pricing is free during beta and planned at $75 per cluster monthly afterward, excluding resource costs.
Logbook - 2022-07-18
This Logbook entry records Fly.io product, platform, documentation, and interface changes on 2022-07-18. Highlights include $99 in usage credits for the Pro Plan, SOC2 compliance, persistent-volume expansion through fly volumes extend, SFTP access to apps over 6PN and WireGuard, and routing changes intended to improve latency. flyctl gained debug JSON formatting, expanded machine-app deployment and status commands, preserved mounted volumes during deployments, and added rolling deploys that wait for each VM to start. The entry also describes fixes for autoscaling, the Fly Redis image, GraphQL validation, Phoenix launches, and dashboard presentation, alongside updated Laravel and persistent-volume documentation and improved Heroku token-failure emails.
Logbook - 2022-06-10
Fly.io’s June 10, 2022 logbook lists platform, dashboard, deployment, and account-management changes for its globally distributed application hosting service. WireGuard peer synchronization now adds only peers whose state changed, while removing a global lock from proxy tracing instrumentation improves TLS handshaking and HTTP parsing under load; the post says these changes should speed userspace features, avoid API timeouts, and reduce response times. Deployments gain build-time secrets through fly deploy’s --build-secret flag, larger rolling-deploy batches that are expected to make multi-region releases 2–5x faster, and restored Turboku webhook redeployment from Heroku. Dashboard updates include intelligent longpolling fallback, visible certificate deletion controls, monitoring links, safer app deletion confirmation, and corrected metrics and coupon visibility, while the Jupyter launcher now requires a credit card after abuse.
Logbook - 2022-05-05
Fly.io’s 2022-05-05 Logbook reports a broad set of infrastructure, product, security, documentation, and bug-fix updates intended to run customer apps faster and more reliably in VMs close to users worldwide. Remote builders now use regional Docker Hub mirrors, clean unused RUN --mount=type=cache build caches, and perform cleanup at startup and shutdown; the new behavior is default for new builders, while existing builders must be destroyed to receive it. WireGuard gateway improvements address load issues, add regional capacity, and support future redundancy, while another IAD gateway is intended to improve connectivity for customer 6pn networks and CI, especially GitHub Actions. The update also adds API security logging for SOC2, re-enables OpenTelemetry, speeds the proxy build pipeline, changes default persistent volume size from 10GB to 3GB, and ships a dashboard, payment retry feature, VSCode fly.toml validation and documentation changes, alongside GraphQL performance work and fixes for monitoring, invoices, encryption, and Homebrew updates.
Our User-Mode WireGuard Year
Fly.io’s flyctl needed to let users SSH into Firecracker VMs and reach private 6PN networks without requiring root access or system-level WireGuard configuration. The solution runs wireguard-go, a userland TCP/IP stack derived from gVisor’s Go implementation, and golang.org/x/crypto/ssh inside flyctl, later coordinating sessions through a cross-platform background agent. The design initially exposed serious operational and usability problems: concurrent sessions displaced one another, agent behavior was fragile, and remote peer creation was slow, while deploys and remote builds expanded the mechanism’s role. The team improved concurrency, Windows support, DNS resolution, configuration handling, and peer-creation latency, and built an unmerged WebSocket path for networks blocking WireGuard’s UDP transport. The author concludes that the complexity was costly early in 2021 but enabled capabilities such as private DNS lookup and ping.
Better Business Intelligence in Elixir with Livebook
The post argues that Elixir developers already perform business intelligence when they use iex or a Rails console to query application data, calculate figures, and produce CSV rows for managers. It presents Livebook as a more capable alternative for young projects that need analysis without adopting a large, continuously running BI application such as Metabase. Because notebooks execute Elixir code and can communicate with running applications through BEAM/OTP clustering, they can reuse business logic, Ecto schemas, associations, and external integrations such as Stripe while avoiding direct database access. Livebook also supplies charts, graphs, tables, Markdown-based sharing, and reproducible reruns, with WireGuard enabling secure connections to apps on Fly.io. The resulting notebooks can answer onboarding, sales, and integration questions while preserving work that may later support an admin dashboard.
Observing Elixir in Production
Observer, included with Elixir and Erlang’s BEAM, is a WxWidgets interface that connects in real time to running nodes, showing process trees and state while allowing processes to be killed. This guide uses Fly.io to connect a local Observer to production nodes through a release cookie, a WireGuard tunnel, and a helper script. The example is a multi-region Tic-Tac-Toe application in Frankfurt and Los Angeles; a local IEx session joins the remote cluster, and Observer can connect to either node. Inspecting a live game exposes its GenServer state in Erlang representation. Killing its process immediately starts a replacement with a different PID and leaves other games running, but recovery still requires the player to reload the page or restart game joining.
SSH and User-mode IP WireGuard
Fly.io’s SSH feature addresses the need to access applications running in Firecracker micro-VMs without requiring users to build containers with an SSH service or install WireGuard. The first part uses 6PN, a private IPv6-based network routed across Fly’s WireGuard mesh, and Hallpass, a small Go SSH server bound to each instance’s 6PN address. Authentication relies on an organization’s root SSH certificate published in private DNS, while the API signs short-lived user certificates. To support users without WireGuard installed, flyctl combines wireguard-go’s userland protocol with gVisor’s user-mode TCP/IP stack, allowing ordinary Go networking code and DNS resolution to operate over the private network. The resulting certificate-based SSH path is built into flyctl and reaches internal DNS names such as dogmatic-potato-342.internal, although the user-mode implementation is substantially slower than kernel TCP/IP.
The January 2021 Fly ChangeLog
The January 2021 Fly ChangeLog covers platform and flyctl changes from December 2020 through January 2021, focusing on private networking, VPN access, scaling, and infrastructure work. 6PN networking adds private communication and DNS-based app discovery over Fly's WireGuard mesh, while IPv6 WireGuard Peering provides tunnels into an organization; the new wireguard command creates, lists, and removes tunnels. Fly also made instance-count scaling the default model, separating it from autoscaling through scale and autoscale commands. Backend updates switched image pulling to containerd, improving subsequent boot times for larger images, added app lifecycle events to application logs, and made volume initialization lazy with encryption enabled by default. The listed flyctl releases also add volume, memory-scaling, region, completion, and WireGuard updates, alongside bug fixes.
Fly In 2020 - A year in features (and articles)
Fly’s 2020 retrospective covers the launch and expansion of its Global Application Platform, alongside features and articles released throughout the year. The platform added Turboku for moving Heroku apps to the edge, Cloud Native Buildpack support, Custom Domains with a GraphQL management API, scaling controls, Deno support, pausing and resuming, persistent volumes, and builtin Flyctl builders for Node, Ruby, Deno, Go, and static websites. Later additions included a new logging system, 6PN Private Networking, WireGuard peering, graceful VM shutdown options, and revised VM sizes, while examples demonstrated services such as Redis, MinIO, Gogs, and MQTT using persistent volumes. The article also records supporting work including GoStatic, JSON output in Flyctl, Firecracker-based workload isolation, and Heroku database usage.
Building clusters with serf, my new favorite thing
The post builds a geographically distributed Headless Chromium cluster on Fly.io and uses it to examine the limits of DNS-based service discovery. It first deploys Chromium in a Singapore Firecracker VM, secures Chrome Debug Protocol access by enabling Fly's private 6PN network and connecting through WireGuard, then scales instances across Singapore, Sydney, Paris, and Chile. Because DNS provides only snapshots and no convenient change notifications, each node runs a leaderless Serf agent bound to its private IPv6 address, tags members by role and region, and joins peers discovered through internal DNS. Serf then supplies an up-to-date membership map, RTT and Vivaldi-based nearness information, plus events and queries; the post concludes that it is simpler than Consul or Zookeeper and more capable than DNS for this setup.
IPv6 WireGuard Peering
Fly.io presents WireGuard Peering as a way to connect applications running on its platform privately to AWS, GCP, or other networks, while preserving automatic communication among apps in an organization. Each organization receives a private IPv6 network called a 6PN, with addresses encoding organization, host, and instance identifiers; BPF enforces isolation, and internal DNS resolves app, regional, and peer names. Because WireGuard’s cryptokey routing does not allow overlapping Allowed IPs, Fly swaps the address’s network and host chunks before and after routing, giving the mesh non-overlapping prefixes without changing IPv6 checksums. Users create regional peer configurations with flyctl wireguard create, receiving /120 delegations and organization-specific DNS. The post says this works for database gateways such as an AWS RDS setup, direct app-instance access, and arbitrary services, with connectivity remaining direct between nodes.
Incoming! 6PN Private Networks
Fly.io introduces 6PN, an IPv6 private network intended to let applications in the same organization communicate without public IP addresses, mTLS, or certificates. Each application instance receives an organization-scoped IPv6 address, while per-application internal DNS zones resolve instances, regions, and sibling apps; enabling it requires private_network = true in fly.toml, though the post says it will soon be on by default. Examples show nginx discovering cache shards, Redis binding to fly-local-6pn, and NATS forming a cross-application cluster through internal names. Behind the feature, addresses encode organization, host, and instance identifiers, while BPF programs enforce network isolation and support WireGuard cryptokey routing; a Rust DNS server serves internal records. The stated goal is simple private connectivity with optional user-managed clustering, service discovery, and certificate infrastructure.
BPF, XDP, Packet Filters and UDP
Deploying unmodified UDP applications across an anycast edge and worker network is harder than proxying HTTP, because UDP carries no convenient metadata for forwarding the original socket addresses. The initial implementation used userland services, AF_PACKET sockets, and custom proxying, but the post argues that kernel-level packet handling is a better fit than adding and monitoring another service. It explains BPF’s evolution into eBPF, then describes using XDP for fast ingress processing, XDP_REDIRECT around WireGuard limitations, and TC BPF on the worker side; Firecracker tap devices make reply traffic appear as host ingress. The verifier requires bounds checks and debugging benefits from dropwatch and perf-based packet tracing. The result is a beta deployment path for UDP applications on Fly.io.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.