---
title: "Distributed Systems"
description: "55 posts about Distributed Systems, summarised, each linking to the original."
---

# Distributed Systems
> 55 posts about Distributed Systems, summarised, each linking to the original.

## Articles

### [Introducing Fly Kubernetes](https://yomu.fyi/post/introducing-fly-kubernetes.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Dec 18, 2023
- Type: Announcement

Fly.io introduces Fly Kubernetes (FKS), an implementation of Kubernetes that runs on its globally distributed container platform for users who want Kubernetes tooling and declarative orchestration. Fly.io says its earlier Nomad-based approach struggled with global scheduling at more than 200,000 applications and with placing workloads in users’ requested regions, so it moved state to individual workers and exposed the Fly Machines API, whose placement requests fail quickly rather than retry elsewhere. FKS combines K3s with Virtual Kubelet, mapping pod lifecycle operations to Fly Machines API calls while running K3s and Virtual Kubelet on one Fly Machine. A generated kubeconfig enables kubectl access, and Pods run on separate Fly Machines as the platform scales the cluster. The design provides declarative replica, autoscaling, and volume definitions plus automatic recovery for failed Machines, while retaining Fly Machines’ trade-off of failing when requested placement is unavailable.


### [Skip the API, Ship Your Database](https://yomu.fyi/post/skip-the-api-ship-your-database.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Sep 13, 2023
- Type: Explainer

The post describes an unexpected use for LiteFS, a distributed SQLite database: replacing service-to-service APIs with read-only database replicas. Typical APIs require anticipating access patterns; shipping the full database lets clients use SQL joins, aggregation, and windowing locally, while one-way LiteFS replication avoids chatty gossip and is simple to configure with an upstream primary hostname. Moving queries to client hardware removes shared service compute contention and the need for tenant rate limiting and timeouts, but replicas cannot handle mutations and raw schema changes create looser contracts, mitigated by backward-compatible changes and views. Data access is harder to restrict in multi-tenant databases, so per-tenant SQLite files are a suggested workaround. The approach is positioned for specific read-heavy use cases, potentially including local copies of Stripe or GitHub data, with event buses such as Kafka generating per-tenant databases.


### [Gossip Glomers](https://yomu.fyi/post/gossip-glomers.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Feb 22, 2023
- Type: Announcement

Gossip Glomers is a series of distributed-systems programming challenges created by Fly.io and Kyle Kingsbury, author of Jepsen, to connect practical coding with distributed-systems theory. Each challenge supplies a workload, a simulated network that can partition or delay messages, and a Jepsen-based verification system; participants write the node binary. The progression covers Echo, a coordination-free unique ID generator, fault-tolerant broadcast, a grow-only counter over Maelstrom’s sequentially consistent key/value store, a Kafka-like replicated log over a linearizable store, and totally available transactions across consistency levels. Initially developed as a resume-blind work-sample evaluation for staff-engineer hiring, the challenges were released publicly because the creators found them too useful and enjoyable to keep internal, with the hardest challenge remaining part of Fly.io’s staff-engineer evaluation.


### [Carving The Scheduler Out Of Our Orchestrator](https://yomu.fyi/post/carving-the-scheduler-out-of-our-orchestrator.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Feb 1, 2023
- Type: Explainer

Fly.io describes why it is carving scheduling out of its existing Nomad-based orchestrator and replacing it with a simpler system centered on flyd and AppsV2. It first reduces orchestration to a client-server supervisor, then explains centralized scheduling: workers register resources, while a server filters candidates by constraints and ranks them. The proposed design avoids jobs sitting pending or being placed in nearby but incorrect regions; requesting VMs in MAD yields MAD capacity or nothing. Policy-heavy behavior such as deployment spares, crash recovery across workers, and regional decisions moves into flyctl and other platform components rather than the scheduler. Fly Machines already exposes APIs to reserve, start, and stop individual VMs, with Nomad still used in January 2023 and flyd expected for deployments in December 2023.


### [WAL Mode in LiteFS](https://yomu.fyi/post/wal-mode-in-litefs.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Jan 4, 2023
- Type: Announcement

LiteFS v0.3.0 adds support for SQLite’s write-ahead log (WAL) alongside the previously supported rollback journal, allowing its distributed file system to replicate transactions from either journaling mode. The post explains how rollback journals and WAL differ, then shows how LiteFS observes filesystem calls, identifies changed pages, and emits a common LTX change-set format; in WAL mode, the final frame’s commit field marks transaction completion. Replica nodes validate LTX files, acquire SQLite-compatible locks, and apply the changes, while the new litefs import command transactionally replaces a database for bootstrapping or snapshot reversion. Version 0.3.0 also replaces previous-page checksum tracking with a per-page checksum map, costing about 3 MB of RAM per gigabyte and roughly 5 ms of CPU time per gigabyte per commit, and adds a trace log whose enabled use can double disk I/O.


### [Launching Redis by Upstash](https://yomu.fyi/post/launching-redis-by-upstash.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Dec 15, 2022
- Type: Announcement

Fly.io and Upstash launched a managed Redis service positioned close to Fly.io applications, with global replicas, a free tier, and private encrypted IPv6 connectivity. The integration uses Flycast, Machines API, WireGuard networking, and PROXY protocol headers: flyctl provisions a Flycast IP, routes it through global proxies to the nearest healthy Redis instance, and lets multitenant Redis identify the customer database without exposing its password. Applications can add replicas and write forwarding through flyctl while retaining a single Redis URI; writes go to the primary and reads and writes use geographically nearest replicas. The post illustrates edge HTML caching for a Rails recipe app, reporting low-millisecond global time to first byte for cached GET requests. It also describes read-your-own-write handling, including a 500ms replication wait and fallback reads from the source of truth.


### [A Foolish Consistency: Consul at Fly.io](https://yomu.fyi/post/a-foolish-consistency-consul-at-fly-io.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Mar 29, 2022
- Type: Explainer

Fly.io’s global application platform uses Consul as a service-discovery catalog, but the post argues that a single globally consistent store is a poor fit for every part of the system. Consul Servers run Raft, while lightweight agents register events; fly-proxy consumes generated JSON describing service instances, and separate data tracks network distance. Because load and health are volatile, Fly.io moved load gossip to NATS and lets fly-proxy generate and distribute health hints, while Consul remains useful for orchestration and status. Strong consistency remains important for DNS, where stale or missing addresses can break ordinary applications; the proposed remedy is optional internal Anycast, routing internal traffic through fly-proxy. The conclusion is qualified: Consul is easy to deploy and valuable for infrastructure configuration, but Fly.io’s trajectory is away from using it as the backing store for a global app platform.


### [32 Bit Real Estate](https://yomu.fyi/post/32-bit-real-estate.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Oct 18, 2021
- Type: Explainer

Fly.io’s globally distributed network needs routable IPv4 addresses because it assigns distinct public addresses to applications, including non-HTTP services such as UDP DNS servers and TCP game servers. The post explains how address control depends on Regional Internet Registry transfers, a BGP4 ASN, upstream peering, and advertising the same address from multiple locations for Anycast routing. IPv6 and TLS SNI can reduce address demand, but IPv6 connectivity remains unreliable for many users and SNI does not cover non-TLS applications; keeping separate provisioning paths is also considered undesirable. IPv4 blocks are therefore bundled with apps, while outright acquisition is preferred to leasing because addresses can remain assigned indefinitely, despite future IPv6 obsolescence and the risk that their value could eventually fall to zero.


### [Last Mile Redis](https://yomu.fyi/post/last-mile-redis.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Jul 16, 2021
- Type: Problem & solution

Last Mile Redis presents a globally distributed caching design aimed at keeping backend responses under the 100-millisecond threshold by placing Redis near users. It argues that regional datasets overlap less than expected, making one Redis server per region a practical alternative to centralized sharding. The design combines standalone regional caches with a Dallas primary Redis and writable replicas configured with replica-read-only no, allowing local writes while replication distributes global changes. A DEL issued at the primary can therefore purge bad cache data across regions, while timestamped keys and expiration handle ordinary freshness; the same mechanism can also broadcast content or fan out queue messages.


### [Globally Distributed Postgres](https://yomu.fyi/post/globally-distributed-postgres.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Jun 30, 2021
- Type: Problem & solution

Globally distributed Postgres describes a way to serve a conventional CRUD application across regions while keeping Postgres writes directed to one primary and reads near users. Fly’s proxy sends each request to the local application instance, where most requests succeed; when a write reaches a read-only replica, Rails catches PG::ReadOnlySqlTransaction and returns a fly-replay header naming PRIMARY\_REGION. The proxy then retries the complete HTTP request inside its network, avoiding per-query cross-region database routing and allowing the application to retain its ordinary connection behavior. The approach can also support session-based read-your-own-writes and replica-freshness checks, but it is not suitable for every workload; the text contrasts it with geographically distributed databases such as CockroachDB.


### [Observing Elixir in Production](https://yomu.fyi/post/observing-elixir-in-production.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Jun 14, 2021
- Type: Tutorial

Observer, included with Elixir and Erlang’s BEAM, is a WxWidgets interface that connects in real time to running nodes, showing process trees and state while allowing processes to be killed. This guide uses Fly.io to connect a local Observer to production nodes through a release cookie, a WireGuard tunnel, and a helper script. The example is a multi-region Tic-Tac-Toe application in Frankfurt and Los Angeles; a local IEx session joins the remote cluster, and Observer can connect to either node. Inspecting a live game exposes its GenServer state in Erlang representation. Killing its process immediately starts a replacement with a different PID and leaves other games running, but recovery still requires the player to reload the page or restart game joining.


### [Building a Distributed Turn-Based Game System in Elixir](https://yomu.fyi/post/building-a-distributed-turn-based-game-system-in-elixir.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Apr 29, 2021
- Type: Tutorial

Mark Ericksen describes building a proof-of-concept distributed, clustered game system for turn-based multiplayer applications with Elixir and Phoenix LiveView. Rather than separate JavaScript and API layers or external state systems such as Kafka and Redis, the design uses server-rendered HTML over websockets, Phoenix.PubSub, and Horde’s distributed process registry. A player’s LiveView sends moves to a GameServer, which publishes updated game state so subscribed LiveViews re-render for every connected player; Tic-Tac-Toe supplies the model. Hosted on Fly.io, geographically close web servers connect over a private WireGuard network while a game’s single GameServer remains the source of truth, with libcluster handling cluster formation and healing. The resulting open-source proof of concept supports multiplayer games, four-letter game codes, on-demand creation, responsive updates, and globally clustered servers.


### [The 5-hour CDN](https://yomu.fyi/post/the-5-hour-cdn.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Mar 16, 2021
- Type: Tutorial

The post presents a five-hour path to building a functional content delivery network as a distributed caching proxy rather than a Google-scale infrastructure project. It defines the architecture as NGINX instances deployed across cities, with DNS rules or Anycast directing users to regions and hash-based load balancing assigning URLs to cache shards. Health checks and multi-location monitoring detect failed or slow regions, while peer caches, origin-access protection, and request coalescing address redundancy, security, and cache-miss spikes. The design can improve cache ratios and reduce origin load, but cross-region proxying can add latency, and distributed invalidation remains unresolved; app-specific workloads may be better served by distributing the application itself. The conclusion is that existing NGINX features and modest scripting make a simple CDN practical, while advanced capabilities such as image optimization, WAF, rate limiting, and bot detection remain possible extensions.


### [Puma 4: Hammering Out H13s—A Debugging Story](https://yomu.fyi/post/puma-4-hammering-out-h13s-a-debugging-story.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Richard Schneeman
- Published: Jul 12, 2019
- Type: Incident / postmortem

Heroku investigated mysterious H13s—“Connection closed” without response—affecting Ruby applications during dyno deployments or scaling, especially at high traffic levels. The failure involved a race between SIGTERM shutting down Puma and the router’s separate system continuing to deliver a few requests, but comparison with other webservers indicated Puma needed to handle the condition. A reproduction used Rack applications that sent SIGTERM to Puma and themselves, plus parallel curl requests, reproducing curl exit code 52 and Heroku H13s. Debugging showed Puma relied on process termination to close accepted sockets; Puma 4 instead closes binder listeners before shutdown, allowing the router to retry connections while preserving responses for in-flight requests.


### [Apache Kafka 0.10 Evaluating Performance in Distributed Systems](https://yomu.fyi/post/apache-kafka-0-10-evaluating-performance-in-distributed-systems.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Tom Crayford
- Published: May 27, 2016
- Type: Benchmark

Testing Apache Kafka 0.10 revealed an apparent 33% throughput decline versus Kafka 0.9, prompting an investigation into whether the regression affected different plans, connection types, message sizes, and workload scales. Heroku’s benchmark system used multiple producer and consumer dynos, while the team applied Brendan Gregg’s USE method to examine resource utilization, saturation, and errors. The diagnosis was network-card saturation: Kafka 0.10 adds an 8-byte timestamp to every message, which at 1.5 million messages per second and replication factor 3 produced 288 Mbps of additional cluster traffic, causing packet drops and retransmits. Reproducing the effect by adding 8 bytes to Kafka 0.9 confirmed the explanation, and the post says most production workloads are unlikely to be affected, though clusters near network limits may be.


[Newer posts](https://yomu.fyi/topic/distributed-systems/page/2.md)