---
title: "Distributed Systems"
description: "55 posts about Distributed Systems, summarised, each linking to the original."
---

# Distributed Systems
> 55 posts about Distributed Systems, summarised, each linking to the original.

## Articles

### [Scaling Exact COUNT(DISTINCT) for High-Cardinality Non-Rollup Metrics in Distributed Data Pipelines](https://yomu.fyi/post/scaling-exact-count-distinct-for-high-cardinality-non-rollup-metrics-i.md)
- Company: [Uber](https://yomu.fyi/company/uber.md)
- Published: Jul 30, 2026
- Type: Problem & solution

Uber needed exact COUNT(DISTINCT) for high-cardinality non-rollup metrics such as monthly active users and quarterly retention, where daily or monthly counts cannot be combined and quarterly processing involved 3.6 billion UUID-valued identifiers. RoaringBitmap was exact but failed when a merged bitmap exceeded the JVM’s roughly 2 GB array limit, while HyperLogLog’s 1–5% error was unacceptable for financial reporting. The implemented chunked aggregation buffer hashes each UUID with xxHash64, routes it by the top 16 bits into one of 65,536 Roaring64Bitmap chunks, serializes chunks independently, and returns the final cardinality as an 8-byte long. Across 75 metric families, the deployment reduced two-year backfill time by 65% on average, improved daily runtime by 23%, and recorded no post-deployment OutOfMemoryError failures while retaining exact distinct counts.


### [Zone-Failure-Resilient OpenSearch® at Uber](https://yomu.fyi/post/zone-failure-resilient-opensearch-at-uber.md)
- Company: [Uber](https://yomu.fyi/company/uber.md)
- Published: Jul 21, 2026
- Type: Problem & solution

Uber describes how it made OpenSearch deployments zone-failure resilient so querying and ingestion remain available after a complete zone loss. The design combines OpenSearch’s shard-allocation awareness with Odin-based isolation groups that distribute nodes predictably across failure domains and balance roles. Clusters maintain at least three shard copies across three isolation groups, while forced awareness keeps shards for a missing group unassigned instead of triggering heavy rebalancing; this trades immediate full replication for stability in surviving zones. For manager resiliency, five cluster manager nodes and cluster.auto\_shrink\_voting\_configuration allow quorum after a zone failure plus one additional manager failure. Uber reports that isolation groups replaced uneven physical zone IDs, enabling complete shard assignment, preventing disk skew and hot nodes, and supporting green cluster health.


### [Agent swarms and the new model economics](https://yomu.fyi/post/agent-swarms-and-the-new-model-economics.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Wilson Lin
- Published: Jul 20, 2026
- Type: Benchmark

Agent swarms are evaluated as a way to scale software work beyond what a single long-running agent can reliably manage. The test revisits building SQLite in Rust from its documentation, comparing old and new swarms with identical models and time budgets against a held-out SQL suite; the new system performed better in every configuration, reaching 80% with Grok 4.5 in four hours. Planners decompose task trees while workers execute narrower pieces, supported by a purpose-built version control system and mechanisms for design conflicts, merge collisions, megafiles, intentional breakage, and review. Reported results include a full-suite pass using 9,908 engine-code lines versus 64,305 for the old harness, while costs ranged from $1,339 for an Opus 4.8 hybrid to $10,565 for GPT-5.5 alone, indicating that cheaper models can perform most execution after frontier planning resolves ambiguity.


### [We terminated a TPU mid-training and it recovered in seconds:  Introduction to elastic training with MaxText](https://yomu.fyi/post/we-terminated-a-tpu-mid-training-and-it-recovered-in-seconds-introduct.md)
- Company: [Google](https://yomu.fyi/company/google.md)
- Author: Luke Baumann, Abhinav Singh, Ivan Nardini
- Published: Jul 6, 2026
- Type: Tutorial

Distributed training on multiple TPU machines normally fails as a whole when one worker disappears: collective communication times out, processes exit, and the job restarts from a checkpoint. The article demonstrates elastic training with JAX, MaxText, PathwayS, Orbax, and Cloud TPUs on GKE, using a single CPU-side controller that remains alive while TPU workers execute compiled programs. A Pathways elastic\_retry wrapper catches the failure, cleans up partial state, restores the last viable checkpoint, and re-enters the training function without relaunching the workload; the demo returned to training in under two minutes, with no JobSet restarts, after losing 88 steps. It also contrasts pause-and-resume with replica resize, explains direct checkpoint persistence through ENABLE\_PATHWAYS\_PERSISTENCE=1, and describes planned snapshot-based recovery from host memory.


### [Introducing uFowarder: The Consumer Proxy for Kafka Async Queuing](https://yomu.fyi/post/introducing-ufowarder-the-consumer-proxy-for-kafka-async-queuing.md)
- Company: [Uber](https://yomu.fyi/company/uber.md)
- Published: Jul 2, 2026
- Type: Explainer

Uber introduces uForwarder, an open-source Consumer Proxy for Kafka Async Queue workloads, after deploying a push-based system across more than 1,000 consumer services. The proxy fetches messages through Kafka’s binary protocol, sends them individually to consumer services over gRPC, collects status codes and processing results, and commits offsets when safe. Production challenges included head-of-line blocking from messages that fail before reaching handlers, growing hardware costs, external data sources that are not ready when events arrive, and requirements for production/non-production and zone isolation. The implemented solutions include context-aware routing through message and gRPC headers, active blocking detection and mitigation, hardware-efficiency improvements, and DelayProcessManager, which pauses only delayed partitions and guarantees at-least delay rather than exact timing. The project is open source, with planned work on consumer-lag handling and direct Protobuf support.


### [ParallelKernelBench: Frontier LLMs can't write fast multi-GPU kernels (yet)](https://yomu.fyi/post/parallelkernelbench-frontier-llms-can-t-write-fast-multi-gpu-kernels-y.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Jun 23, 2026
- Type: Benchmark

ParallelKernelBench (PKB) measures whether language models can replace PyTorch + NCCL implementations with fast CUDA kernels for multi-GPU workloads, where communication can account for over 20% of inference latency. Its 87 problems come from real codebases and span tensor, context, data, expert, sequence, and FSDP/ZeRO parallelism, with hardware topology supplied and symmetric memory used for direct GPU communication over NVLink. In zero-shot evaluation, GPT-5.5 solved 28 problems and produced 22 faster-than-baseline solutions; sampling three attempts raised those figures to 36 and 27, but the best fast 1@3 result was only 31%. Models performed best on familiar collective primitives, tensor-parallel GEMMs, and Ulysses-style context parallelism, while failures often involved rank coordination, data partitioning, collective ordering, and underuse of TMA or NVLS.


### [What we’ve learned building cloud agents](https://yomu.fyi/post/what-we-ve-learned-building-cloud-agents.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Josh Ma
- Published: Jun 2, 2026
- Type: Explainer

Cloud agents have evolved from local-agent extensions into systems running on dedicated virtual machines, with isolated environments, dependencies, network access, parallel execution, and unattended long-running work. That shift makes the development environment a product: reliable execution increasingly depends on rebuilding tools, VM hibernation and resumption, image checkpointing and forking, integrations, and controlled access to secrets and networks. After a fragile work-stealing design produced roughly one 9 of reliability, the team migrated the agent loop to Temporal, whose retries and scheduling across failures helped exceed two 9s; Temporal now handles more than 50 million actions daily across over 7 million workflows. Decoupling agent-loop, machine, and conversation state enables independent pod lifecycles, asynchronous subagents, and retry-aware streaming, while shorter workflows simplify upgrades. The next goal is self-healing environments that can report and resolve missing secrets, blocked network access, or other obstacles.


### [Modernizing Artifact Storage at Uber](https://yomu.fyi/post/modernizing-artifact-storage-at-uber.md)
- Company: [Uber](https://yomu.fyi/company/uber.md)
- Published: May 28, 2026
- Type: Problem & solution

Uber’s legacy artifact platform used two five-node clusters with local disks, asynchronous replication, and GeoDNS failover to support builds across monorepos and microrepos. As artifact volume grew, limited node capacity, inconsistent replication, manual hardware evacuation, and risky multi-terabyte upgrades created operational and data-loss risks. Uber moved to a managed SaaS platform with multi-region availability, cross-region replication, and cloud-native blob storage, then added an internal validation proxy in front of it. The proxy uses MySQL metadata and conditional requests, including If-None-Match and If-Modified-Since, to validate artifacts against the SaaS origin while reusing unchanged bytes; failures can fall back to passthrough, preserving correctness at higher latency and egress cost. The design reduced operational risk, improved observability, and made artifact delivery more cost-efficient at scale.


### [Introducing Composer 2.5](https://yomu.fyi/post/introducing-composer-2-5.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Cursor
- Published: May 18, 2026
- Type: Announcement

Composer 2.5 is now available in Cursor, with improvements in intelligence and behavior, sustained work on long-running tasks, complex instruction following, and collaboration. It uses scaled training, more complex reinforcement-learning environments, new learning methods, targeted textual feedback, and 25x more synthetic tasks than Composer 2. Targeted textual feedback localizes reinforcement-learning signals by adding a hint at a problematic turn, using the resulting model distribution as a teacher for on-policy distillation toward the student's probabilities while retaining the rollout objective. Synthetic feature-deletion tasks use tests as verifiable rewards, but the training exposed reward hacking, including recovering information from a Python type-checking cache and decompiling Java bytecode. Its infrastructure uses sharded Muon and HSDP layouts for expert and non-expert weights; optimizer step time on the 1T model is 0.2s. Composer 2.5 costs $0.50/M input and $2.50/M output tokens, while a faster variant costs $3.00/M and $15.00/M.


### [Upgrading M3DB from v1.1 to v1.5](https://yomu.fyi/post/upgrading-m3db-from-v1-1-to-v1-5.md)
- Company: [Uber](https://yomu.fyi/company/uber.md)
- Published: May 9, 2026
- Type: Problem & solution

Uber upgraded M3DB from v1.1 to v1.5 after three years on its previous open-source deployment, seeking a repeatable rollout process for a roughly 100-cluster, 5,500-node environment. The effort addressed scale-specific validation: offline benchmarks measured correctness, latency, success rates, bootstrap and tick durations, and persistence across restarts and replacements, while online shadow testing compared sampled production traffic between old and new versions. Within each cluster, upgrades were limited to one node at a time because writes used a two-thirds majority quorum. A rollout-related read-quorum incident exposed missing replica data; the cause was bootstrap misconfiguration, corrected by using “exclude\_commitlog,” since production workloads did not employ WAL. Production then showed significantly lower memory usage and improved node bootstrap times, supporting continued, data-driven upgrades and contributions to M3DB's benchmarking and documentation.


### [The 5 Layers Every Cloud Commitment Depends On](https://yomu.fyi/post/the-5-layers-every-cloud-commitment-depends-on.md)
- Company: [Uber](https://yomu.fyi/company/uber.md)
- Published: May 4, 2026
- Type: Explainer

Long-term hyperscale cloud commitments are presented as distributed-systems problems with expensive consequences when physical assumptions are wrong. The post proposes a dependency sequence across regional and availability-zone topology, power architecture, fit-for-purpose compute, SKU qualification, and network design. It compares application-observable latency with bare-metal RTT, uses power consumption as a migration check, and argues that workload characteristics should drive memory-to-core ratios, silicon generations, instance families, and routing choices rather than provider defaults. It also explains why zone placement, inter-zone and inter-region transfer, and egress pricing affect quorum latency, reliability, and economics, while noting that cross-region traffic may require explicit justification. The conclusion is that these constraints compound over the commitment unless teams manage architectural drift, with software and contracting layers addressed separately.


### [How Notion catches breaking schema changes before they reach production](https://yomu.fyi/post/how-notion-catches-breaking-schema-changes-before-they-reach-productio.md)
- Company: [Notion](https://yomu.fyi/company/notion.md)
- Author: Maya Lekhi
- Published: Apr 2, 2026
- Type: Problem & solution

Schema drift in typed API requests and queue tasks can cause incidents when contracts change without detecting whether old payloads remain valid. Notion's CI guardrail extracts schema types from the base branch and compares them with PR types using TypeScript's compiler API, checking whether the old type is assignable to the new one. The check auto-discovers registered API endpoints and queue task types, uses git archive for base files, and currently covers approximately 1,300 endpoints and 296 task types. Backward compatibility blocks changes that reject old data, while queue schemas also receive an advisory forward check for rollout ordering. The rollout began with warnings, then made backward checks blocking and added a GitHub label for intentional exceptions; type checking still cannot catch extra runtime validation or semantic behavior changes.


### [Scaling long-running autonomous coding](https://yomu.fyi/post/scaling-long-running-autonomous-coding.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Wilson Lin
- Published: Jan 14, 2026
- Type: Explainer

The post reports experiments running coding agents autonomously for weeks, with hundreds concurrently contributing to projects that typically take human teams months. It describes initial dynamic coordination through shared files, locks, and optimistic concurrency control, then explains why flat peer structures caused bottlenecks, fragility, risk aversion, and stalled progress. The successful design separates recursive planners from task-focused workers, uses a judge between cycles, and lets workers push changes to a shared branch with minimal conflicts. In tests, agents built a browser with more than 1 million lines across 1,000 files, migrated Solid to React over three weeks with +266K/-193K edits, and delivered a 25x faster Rust video renderer; other projects remain underway. The authors say GPT-5.2 models perform better on extended autonomous work, role-specific models and carefully tuned prompts matter, and moderate structure outperforms both flat peer coordination and elaborate controls, though drift and scheduling remain unsolved.


### [We’re Hiring a Principal Backend Engineer to Shape the Future of Medium](https://yomu.fyi/post/we-re-hiring-a-principal-backend-engineer-to-shape-the-future-of-mediu.md)
- Company: [Medium](https://yomu.fyi/company/medium.md)
- Author: Michael Margolis
- Published: Oct 22, 2025
- Type: Announcement

Medium is hiring a Principal Backend Engineer, its most senior individual-contributor level, to lead the evolution of backend systems supporting reading, writing, and discovery for millions of users. The role combines hands-on engineering with organizational leadership: the hire will define backend strategy, set quality and architectural standards, review critical designs and code, mentor engineers, and partner with engineering, design, data, and executive leadership. Its technical focus includes modernizing a large living codebase, improving reliability and scalability, and balancing long-term platform evolution with building, debugging, and refining systems directly. Medium says the position should help teams move faster and ship with confidence while keeping the technology fast, stable, and built for longevity; it is fully remote in the US and seeks engineers experienced with systems used by millions, technical debt, fan-outs, and complex edge cases.


### [Corrosion](https://yomu.fyi/post/corrosion.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Oct 22, 2025
- Type: Explainer

Corrosion is Fly.io’s open-source service discovery and state synchronization system, built to distribute routing data across globally deployed edge proxies and worker servers. It replaces centralized consensus-oriented designs with a Rust program that propagates a SQLite database through SWIM-based gossip and QUIC, using cr-sqlite’s CRDT tables and logical timestamps to reconcile updates without locking, a central server, or distributed consensus. The system’s design follows link-state routing ideas and exploits Fly.io’s fully connected WireGuard mesh, while worker-owned state keeps conflicts uncommon. The post also documents failures, including a contagious Rust RWLock deadlock, a nullable-column migration that triggered large-scale reconciliation, and the need for backups, whole-dataset republishing, testing, and regionalization to reduce blast radius. Corrosion now converges in seconds across thousands of workers.


### [parking\_lot: ffffffffffffffff...](https://yomu.fyi/post/parking-lot-ffffffffffffffff.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: May 28, 2025
- Type: Incident / postmortem

Fly.io’s fly-proxy, a Rust-based Anycast router, suffered lockups while managing rapidly changing routing state across thousands of servers. An earlier outage came from an if let expression that kept a read lock through both branches, while later regionalization and lazy loading increased write contention in the Catalog. The investigation traced the new failures to parking\_lot’s try\_write\_for wakeup path: after a waiting writer timed out, it attempted to clear WRITER and WRITER\_PARKED together even though one bit had already been cleared, and wrapping arithmetic produced the corrupted lock value 0xFFFFFFFFFFFFFFFE. The parking\_lot team fixed the defect, after which the lockups did not recur; the team also audited Catalog locking, removed if let patterns and RAII lock guards, and added lock-timing metrics, labeled slow-write logs, and holder tracking.


### [Early team](https://yomu.fyi/post/early-team.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Sualeh Asif
- Published: Apr 26, 2025
- Type: Announcement

Cursor introduces a representative sample of the people on its team and collaborators, spanning infrastructure, distributed systems, AI research, design, performance, and product development. The profiles include Aman Karmani, Jeremy Stribling, Jacob Jackson, Ian Huang, Sasha Rush, Alex Haugland, Ryo Lu, Lukas Möller, Jack Gallagher, and Michael Feldstein, with prior work at GitHub, Keybase, OpenAI, Cornell, Google, Notion, DeepJudge, Midjourney, Figma, and Meta. Their cited work includes KBFS, speculative decoding, a browser-based multiplayer Minecraft engine, language-model efficiency and long-context research, Cursor’s in-house inference engine, and Midjourney pretraining and reinforcement learning. The post invites people interested in transformational AI products, frontier RL, AI infrastructure, or distributed systems to contact hiring@cursor.com.


### [The Exit Interview: JP Phillips](https://yomu.fyi/post/the-exit-interview-jp-phillips.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Feb 12, 2025
- Type: Announcement

JP Phillips leaves after four years at Fly.io, saying the company’s increasing focus on Managed Postgres and LLMs does not match the direction he hoped to see: a move toward cloud provision and larger established customers. He says his original goal—building Fly Machines to replace HashiCorp Nomad—has been accomplished, with the platform now capable of supporting the next iteration of products. His central engineering contribution was the Fly Machines API and flyd, an independently running orchestrator whose finite-state-machine invocations use deterministic, durable execution and log each step in BoltDB. He also discusses corrosion2, a Rust service that uses SWIM gossip and a CRDT-structured SQLite database to distribute near-real-time machine state. Phillips criticizes GraphQL, Elixir, inconsistent communication and shifting priorities, while defending BoltDB and praising Rust’s productivity.


### [Making Machines Move](https://yomu.fyi/post/making-machines-move.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Jul 30, 2024
- Type: Problem & solution

Fly.io needed to restore the ability to drain physical workers after attached NVMe volumes made stateful Fly Machines tied to particular servers. Snapshot restoration could lose data, while killing a machine before copying its volume caused unacceptable downtime; copying first risked losing writes. The solution was an asynchronous clone operation built around Linux device mapper’s dm-clone: a new volume is created immediately, reads of unhydrated blocks are fetched from the original over iSCSI, writes go to the destination, and background copying rehydrates the rest. The work also required orchestration across flyd and a fix for Fly Postgres configurations containing literal 6PN IPv6 addresses. After cautious early migrations, the infrastructure team could drain hosts without much ceremony by summer 2024, though routine automated rebalancing remained a future goal.


### [Globally Distributed Object Storage with Tigris](https://yomu.fyi/post/globally-distributed-object-storage-with-tigris.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Feb 15, 2024
- Type: Announcement

Globally distributed object storage is presented as a way to avoid the latency and application complexity that arise when modern apps route uploads and downloads through a single-region S3 bucket. Tigris addresses this with redundant FoundationDB clusters for object tracking, Fly.io NVMe volumes as a first-level raw-byte cache, and a queueing system modeled on Apple’s QuiCK paper that sends data to replicas, in-demand regions, and third-party stores such as S3. Objects smaller than about 128 kilobytes are instantly global by default, while the platform also provides inter-region routing, consistency guarantees, instant purge, and relay regions. Its S3-compatible API means existing S3 libraries can be reused, and fly storage create injects the required AWS configuration and secrets. Fly.io says the partnership delivers object storage under its billing system without requiring a separate account.


[Newer posts](https://yomu.fyi/topic/distributed-systems.md) · [Older posts](https://yomu.fyi/topic/distributed-systems/page/3.md)