Building clusters with serf, my new favorite thing

Summary

The post builds a geographically distributed Headless Chromium cluster on Fly.io and uses it to examine the limits of DNS-based service discovery. It first deploys Chromium in a Singapore Firecracker VM, secures Chrome Debug Protocol access by enabling Fly's private 6PN network and connecting through WireGuard, then scales instances across Singapore, Sydney, Paris, and Chile. Because DNS provides only snapshots and no convenient change notifications, each node runs a leaderless Serf agent bound to its private IPv6 address, tags members by role and region, and joins peers discovered through internal DNS. Serf then supplies an up-to-date membership map, RTT and Vivaldi-based nearness information, plus events and queries; the post concludes that it is simpler than Consul or Zookeeper and more capable than DNS for this setup.

Takeaways

1. 1

   Fly.io can deploy Headless Chromium as a Firecracker VM and scale instances across Singapore, Sydney, Paris, and Chile.

2. 2

   Serf agents use private IPv6 addresses, WAN mode, and role and region tags to maintain a leaderless membership view across nodes.

3. 3

   Serf's Vivaldi implementation provides relative network-nearness estimates, while its events and queries support cluster-wide messaging and responses.