A Foolish Consistency: Consul at Fly.io

Summary

Fly.io’s global application platform uses Consul as a service-discovery catalog, but the post argues that a single globally consistent store is a poor fit for every part of the system. Consul Servers run Raft, while lightweight agents register events; fly-proxy consumes generated JSON describing service instances, and separate data tracks network distance. Because load and health are volatile, Fly.io moved load gossip to NATS and lets fly-proxy generate and distribute health hints, while Consul remains useful for orchestration and status. Strong consistency remains important for DNS, where stale or missing addresses can break ordinary applications; the proposed remedy is optional internal Anycast, routing internal traffic through fly-proxy. The conclusion is qualified: Consul is easy to deploy and valuable for infrastructure configuration, but Fly.io’s trajectory is away from using it as the backing store for a global app platform.

Takeaways

1. 1

   Consul Servers use the Raft consensus protocol, while Consul Agents relay events such as service-instance termination and registration.

2. 2

   Fly.io replaced Consul-based load tracking with NATS because load information is rapidly changing and needs fresh hints rather than a consensus-backed database.

3. 3

   The planned internal Anycast feature provides stable addresses that route through fly-proxy, extending Internet-traffic routing techniques to internal Postgres and Redis connectivity.