Loading…
Topic
53 posts about Distributed Systems. Every summary links to the original.
See the idea
A tradeoff view of blockchain: a shared ledger buys verifiable settlement between parties that do not share an operator, and charges throughput, latency, and governance in return.
A Bloom filter is a small bit array plus a few hash functions that answers "has this been seen?" with no false negatives and a false-positive rate you choose up front.
Backpressure is a feedback loop that slows work when downstream capacity or health deteriorates. Compare its gains, costs, and signals for choosing a control point.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Gossip Glomers is a series of distributed-systems programming challenges created by Fly.io and Kyle Kingsbury, author of Jepsen, to connect practical coding with distributed-systems theory. Each challenge supplies a workload, a simulated network that can partition or delay messages, and a Jepsen-based verification system; participants write the node binary. The progression covers Echo, a coordination-free unique ID generator, fault-tolerant broadcast, a grow-only counter over Maelstrom’s sequentially consistent key/value store, a Kafka-like replicated log over a linearizable store, and totally available transactions across consistency levels. Initially developed as a resume-blind work-sample evaluation for staff-engineer hiring, the challenges were released publicly because the creators found them too useful and enjoyable to keep internal, with the hardest challenge remaining part of Fly.io’s staff-engineer evaluation.
Fly.io describes why it is carving scheduling out of its existing Nomad-based orchestrator and replacing it with a simpler system centered on flyd and AppsV2. It first reduces orchestration to a client-server supervisor, then explains centralized scheduling: workers register resources, while a server filters candidates by constraints and ranks them. The proposed design avoids jobs sitting pending or being placed in nearby but incorrect regions; requesting VMs in MAD yields MAD capacity or nothing. Policy-heavy behavior such as deployment spares, crash recovery across workers, and regional decisions moves into flyctl and other platform components rather than the scheduler. Fly Machines already exposes APIs to reserve, start, and stop individual VMs, with Nomad still used in January 2023 and flyd expected for deployments in December 2023.
LiteFS v0.3.0 adds support for SQLite’s write-ahead log (WAL) alongside the previously supported rollback journal, allowing its distributed file system to replicate transactions from either journaling mode. The post explains how rollback journals and WAL differ, then shows how LiteFS observes filesystem calls, identifies changed pages, and emits a common LTX change-set format; in WAL mode, the final frame’s commit field marks transaction completion. Replica nodes validate LTX files, acquire SQLite-compatible locks, and apply the changes, while the new litefs import command transactionally replaces a database for bootstrapping or snapshot reversion. Version 0.3.0 also replaces previous-page checksum tracking with a per-page checksum map, costing about 3 MB of RAM per gigabyte and roughly 5 ms of CPU time per gigabyte per commit, and adds a trace log whose enabled use can double disk I/O.
Fly.io and Upstash launched a managed Redis service positioned close to Fly.io applications, with global replicas, a free tier, and private encrypted IPv6 connectivity. The integration uses Flycast, Machines API, WireGuard networking, and PROXY protocol headers: flyctl provisions a Flycast IP, routes it through global proxies to the nearest healthy Redis instance, and lets multitenant Redis identify the customer database without exposing its password. Applications can add replicas and write forwarding through flyctl while retaining a single Redis URI; writes go to the primary and reads and writes use geographically nearest replicas. The post illustrates edge HTML caching for a Rails recipe app, reporting low-millisecond global time to first byte for cached GET requests. It also describes read-your-own-write handling, including a 500ms replication wait and fallback reads from the source of truth.
Fly.io’s global application platform uses Consul as a service-discovery catalog, but the post argues that a single globally consistent store is a poor fit for every part of the system. Consul Servers run Raft, while lightweight agents register events; fly-proxy consumes generated JSON describing service instances, and separate data tracks network distance. Because load and health are volatile, Fly.io moved load gossip to NATS and lets fly-proxy generate and distribute health hints, while Consul remains useful for orchestration and status. Strong consistency remains important for DNS, where stale or missing addresses can break ordinary applications; the proposed remedy is optional internal Anycast, routing internal traffic through fly-proxy. The conclusion is qualified: Consul is easy to deploy and valuable for infrastructure configuration, but Fly.io’s trajectory is away from using it as the backing store for a global app platform.
Fly.io’s globally distributed network needs routable IPv4 addresses because it assigns distinct public addresses to applications, including non-HTTP services such as UDP DNS servers and TCP game servers. The post explains how address control depends on Regional Internet Registry transfers, a BGP4 ASN, upstream peering, and advertising the same address from multiple locations for Anycast routing. IPv6 and TLS SNI can reduce address demand, but IPv6 connectivity remains unreliable for many users and SNI does not cover non-TLS applications; keeping separate provisioning paths is also considered undesirable. IPv4 blocks are therefore bundled with apps, while outright acquisition is preferred to leasing because addresses can remain assigned indefinitely, despite future IPv6 obsolescence and the risk that their value could eventually fall to zero.
Last Mile Redis presents a globally distributed caching design aimed at keeping backend responses under the 100-millisecond threshold by placing Redis near users. It argues that regional datasets overlap less than expected, making one Redis server per region a practical alternative to centralized sharding. The design combines standalone regional caches with a Dallas primary Redis and writable replicas configured with replica-read-only no, allowing local writes while replication distributes global changes. A DEL issued at the primary can therefore purge bad cache data across regions, while timestamped keys and expiration handle ordinary freshness; the same mechanism can also broadcast content or fan out queue messages.
Globally distributed Postgres describes a way to serve a conventional CRUD application across regions while keeping Postgres writes directed to one primary and reads near users. Fly’s proxy sends each request to the local application instance, where most requests succeed; when a write reaches a read-only replica, Rails catches PG::ReadOnlySqlTransaction and returns a fly-replay header naming PRIMARY_REGION. The proxy then retries the complete HTTP request inside its network, avoiding per-query cross-region database routing and allowing the application to retain its ordinary connection behavior. The approach can also support session-based read-your-own-writes and replica-freshness checks, but it is not suitable for every workload; the text contrasts it with geographically distributed databases such as CockroachDB.
Observer, included with Elixir and Erlang’s BEAM, is a WxWidgets interface that connects in real time to running nodes, showing process trees and state while allowing processes to be killed. This guide uses Fly.io to connect a local Observer to production nodes through a release cookie, a WireGuard tunnel, and a helper script. The example is a multi-region Tic-Tac-Toe application in Frankfurt and Los Angeles; a local IEx session joins the remote cluster, and Observer can connect to either node. Inspecting a live game exposes its GenServer state in Erlang representation. Killing its process immediately starts a replacement with a different PID and leaves other games running, but recovery still requires the player to reload the page or restart game joining.
Mark Ericksen describes building a proof-of-concept distributed, clustered game system for turn-based multiplayer applications with Elixir and Phoenix LiveView. Rather than separate JavaScript and API layers or external state systems such as Kafka and Redis, the design uses server-rendered HTML over websockets, Phoenix.PubSub, and Horde’s distributed process registry. A player’s LiveView sends moves to a GameServer, which publishes updated game state so subscribed LiveViews re-render for every connected player; Tic-Tac-Toe supplies the model. Hosted on Fly.io, geographically close web servers connect over a private WireGuard network while a game’s single GameServer remains the source of truth, with libcluster handling cluster formation and healing. The resulting open-source proof of concept supports multiplayer games, four-letter game codes, on-demand creation, responsive updates, and globally clustered servers.
The post presents a five-hour path to building a functional content delivery network as a distributed caching proxy rather than a Google-scale infrastructure project. It defines the architecture as NGINX instances deployed across cities, with DNS rules or Anycast directing users to regions and hash-based load balancing assigning URLs to cache shards. Health checks and multi-location monitoring detect failed or slow regions, while peer caches, origin-access protection, and request coalescing address redundancy, security, and cache-miss spikes. The design can improve cache ratios and reduce origin load, but cross-region proxying can add latency, and distributed invalidation remains unresolved; app-specific workloads may be better served by distributing the application itself. The conclusion is that existing NGINX features and modest scripting make a simple CDN practical, while advanced capabilities such as image optimization, WAF, rate limiting, and bot detection remain possible extensions.
Heroku investigated mysterious H13s—“Connection closed” without response—affecting Ruby applications during dyno deployments or scaling, especially at high traffic levels. The failure involved a race between SIGTERM shutting down Puma and the router’s separate system continuing to deliver a few requests, but comparison with other webservers indicated Puma needed to handle the condition. A reproduction used Rack applications that sent SIGTERM to Puma and themselves, plus parallel curl requests, reproducing curl exit code 52 and Heroku H13s. Debugging showed Puma relied on process termination to close accepted sockets; Puma 4 instead closes binder listeners before shutdown, allowing the router to retry connections while preserving responses for in-flight requests.
Testing Apache Kafka 0.10 revealed an apparent 33% throughput decline versus Kafka 0.9, prompting an investigation into whether the regression affected different plans, connection types, message sizes, and workload scales. Heroku’s benchmark system used multiple producer and consumer dynos, while the team applied Brendan Gregg’s USE method to examine resource utilization, saturation, and errors. The diagnosis was network-card saturation: Kafka 0.10 adds an 8-byte timestamp to every message, which at 1.5 million messages per second and replication factor 3 produced 288 Mbps of additional cluster traffic, causing packet drops and retransmits. Reproducing the effect by adding 8 bytes to Kafka 0.9 confirmed the explanation, and the post says most production workloads are unlikely to be affected, though clusters near network limits may be.
The best engineering reads, once a week.
