Loading…
Topic
55 posts about Distributed Systems. Every summary links to the original.
See the idea
A tradeoff view of blockchain: a shared ledger buys verifiable settlement between parties that do not share an operator, and charges throughput, latency, and governance in return.
A Bloom filter is a small bit array plus a few hash functions that answers "has this been seen?" with no false negatives and a false-positive rate you choose up front.
Backpressure is a feedback loop that slows work when downstream capacity or health deteriorates. Compare its gains, costs, and signals for choosing a control point.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Minions is a dynamic, distributed virtual GPU cloud that aggregates idle Apple Silicon GPUs across macOS devices to run large-scale machine learning inference. The work responds to limits in centralized GPU infrastructure, including high cost, limited elasticity, and long wait times when resources are constrained. Demand for GPU-accelerated machine learning inference keeps rising, while modern Apple Silicon macOS devices carry powerful GPUs that stay largely idle during normal usage. Minions offers flexible GPU inference task execution and provides fault tolerance and efficient resource utilization, letting teams use existing hardware in place of additional GPU procurement. The authors report that idle macOS devices can become a practical distributed GPU inference platform in environments where centralized GPU resources are constrained.
The Prime-Sentinel Command (PSC) architecture is an object-oriented master-agent pattern for coordinating edge diagnostic nodes via a centralized orchestrator. Many diagnostic setups tightly couple data polling with central processing, creating bottlenecks and complicating retry logic. PSC isolates agent-level diagnostics into self-contained SentinelProgram instances that sample CPU load, memory utilization, and network state flags, returning structured telemetry payloads. The PrimeProgram controller manages lifecycle dispatch, dynamic registration, batch execution passes, and reporting thresholds, using synchronous list comprehension for sweeps with notes on swapping to concurrent.futures.ThreadPoolExecutor or asyncio.gather for higher node counts. The reference implementation demonstrates three sentinels reporting metrics like CPU 24.15%, RAM 45.20%, and network status STABLE, with payloads designed for JSON serialization across WebSockets, gRPC, or message brokers.
Red Hat OpenShift AI 3.4 introduces Kubeflow Trainer v2, a unified training API that replaces framework-specific CRDs like PyTorchJob and TFJob with a single TrainJob resource. Under the hood, each TrainJob is translated into a JobSet, a Kubernetes-native API for managing groups of Jobs as a single unit, which handles headless services for pod discovery and automatic failure recovery. The post validates the stack through two phases: manual validation with a minimal NCCL all-reduce test across 2 nodes, and real-world training of an embedding-based neural network on a fraud detection dataset using 1, 2, and 4 NVIDIA A10G GPUs. Benchmarking shows near-linear Distributed Data Parallel scaling at 95% efficiency, successful checkpoint recovery after pod kills, and GPU utilization confirmed via DCGM Exporter. The TrainJob API simplifies distributed training by automatically configuring torchrun, environment variables, and pod coordination, while raw JobSets offer more control for multi-stage pipelines and differentiated failure handling.
Salesforce’s Cloud Atlas team operates a globally distributed identity data store that must maintain five nines of availability, roughly five minutes of downtime per year. The team redesigned service protection after per-instance rate limiting proved insufficient under autoscaling and increasingly bursty, unpredictable agent-driven traffic. The new approach combines intelligent load shedding, which monitors queue time as an early indicator of system pressure before error rates spike, with coordinator-free global quota management. Servers independently measure and share customer usage to throttle noisy tenants without a centralized coordinator, preventing one customer’s surge from affecting others. This design prevents cascading failures by shedding lower-priority work before overload spreads across dependent services, ensuring graceful degradation rather than widespread outage.
Notion replaced its text-editing system because its former last-write-wins storage could overwrite concurrent edits to the same block, with Offline Mode increasing the risk of lost changes. The new system uses a Conflict-free Replicated Data Type based on Replicated Growable Array, representing inserted text as uniquely identified items, retaining deletions as tombstones, and deterministically ordering concurrent inserts. Peritext-based annotation operations merge rich-text formatting such as bold, italics, and hyperlinks while preserving extendability rules. To keep edits correctly positioned when Enter splits a block, Notion introduced text slices, text slice trees, text instances, and compact search labels for locating slices across blocks. Deployed in July 2025, the system processes millions of CRDT operations per minute and supports Offline Mode and agent collaboration.
A cloud-based database management system stores, manages, and queries data on third-party cloud infrastructure instead of local or on-premises servers. It retains capabilities such as query optimization, security, transactions, and support for relational SQL, NoSQL, data warehouse, and in-memory workloads, while operating across distributed nodes. Organizations can self-manage database software on cloud virtual machines or use DBaaS, where the provider manages the software and infrastructure; connections typically use SQL, APIs, or web consoles. Virtualized resources can scale, and providers may handle patching, backups, maintenance, and failover, while replication across availability zones supports recovery and availability. The comparison with on-premises systems covers ownership, scalability, cost models, maintenance, accessibility, control, compliance, total cost, and emerging capabilities including AI integration, vector search, autonomous operation, multicloud, hybrid, and serverless architectures.
Canva’s queue workers were designed to consume messages greedily, but that behavior could amplify dependency failures, waste scarce capacity, fill dead-letter queues, and create on-call work. Worker Backpressure adds a local feedback loop to the shared queue library: each processed message reports success or failure, a pluggable controller maintains a 0.0-to-1.0 backoff factor against a configured failure-rate set point, and permits scale concurrency at each poll. The mechanism requires no external coordinator or network calls and adds two arithmetic operations to the runtime path, while trading some throughput for failure containment. In two production incidents, it reduced worker pressure during errors and recovered automatically; one sustained-overload incident recorded a 2.76% average failure rate, 22 DLQ messages over 32.5 hours, and no paging, while rollout to all queue workers remained the immediate goal.
Uber describes an automated method for cataloging dependency behavior across its service mesh, where nested asynchronous calls make failure attribution difficult. Because distributed traces are sampled and erroneous requests are rare, trace-based correlation can take too long; the system instead instruments yarpc middleware to observe every request and correlate inbound failures with outbound failures at service-and-endpoint granularity. An in-memory request record links each inbound request to its outbound calls, while emitted metrics capture caller and callee nodes and whether each errored. The resulting edge probability, calculated as caller failures divided by requests where the callee failed, classifies dependencies as fail-close at Pc >= 0.8, fail-open at Pc <= 0.2, or unknown between those thresholds. Uber says the initiative cataloged dependencies of critical APIs and indexed most fail-close anomalies, supporting reliability work and regression guardrails for Failover Architecture 2.0.
ClickHouse’s replica-aware routing is now in Public Beta for Enterprise customers, addressing cases where temporary tables and named sessions become invisible after a request reaches another replica. The feature routes requests to a consistent replica using an X-ClickHouse-Replica-Tag or a native TLS SNI override, supporting session-scoped access and read-after-write consistency. Over HTTP, Envoy reads the header and uses consistent hashing; native connections decouple the SNI used for routing from the hostname validated against the certificate. Stickiness is best-effort and can change after upgrades, restarts, or scaling, so temporary objects and session settings may need recreation. The feature is available through service settings for Enterprise plans on standard ClickHouse Cloud and BYOC.
Pinterest describes how its Manas distributed search platform is evolving embedding retrieval for a corpus growing toward tens of billions of embeddings. Traditional HNSW indexes require all data in RAM, while single-vector two-tower retrieval limits context-dependent relevance, motivating changes across quantization, SSD-based serving, and multi-embedding retrieval. Manas uses Scalar Quantization and Product Quantization across HNSW and IVF, with SQ preserving over 90% recall while reducing index size, and Linear Scaling plus SIMD cuts per-query computing resources by 10–15%. For disk-based serving, SPANN with PQ is reported as 4.5x faster than plain SPANN and over 3x DiskANN QPS, while a five-billion-embedding production evaluation saved over 40% CPU time with under 5% recall drop. The platform is also adding ColBERT-style late interaction using Sum of MaxSim, with a pilot in progress.
ClickHouse Cloud’s On-Demand Compute is announced in private preview as a way to run eligible queries on a shared pool of workers outside a service’s primary cluster, avoiding production contention and reactive autoscaling delays. It uses ClickHouse’s multistage distributed query execution and cost-based optimizer to distribute complex workloads, with queries requesting worker counts through settings including make_distributed_plan and distributed_plan_workers_num. Workers are leased for at least 60 seconds, reused during the lease, and released when it expires; concurrent queries share the pool, which grows to the largest request. In a stated benchmark, On-Demand Compute was faster for the tested workload and at 25 concurrent queries delivered similar performance while using 50% less compute, though results vary by query shape.
An AsyncGRPO setup uses LoRA adapter-only synchronization to separate vLLM Jobs, rather than moving full model weights over NCCL. A Storage Bucket mounted at the same path lets the trainer publish versioned adapters that replicas load, while a proxy adds authentication, routes rollouts toward KV-cached prefixes, and broadcasts adapter updates. Versioned adapter names prevent cached prefixes from mixing policy versions, and max_staleness=4 requires six LoRA slots during replacement. Five runs identify client-side concurrency, checkpointing, and packing as important throughput factors: raising in-flight requests to 384, disabling checkpointing, and packing improved median step time from 22.9 to 4.8 seconds. At 500 steps, the final run took 53 minutes instead of 3 hours 27 minutes, trained 31% more samples, and had basically the same reward curve.
Uber’s failover redesign exposed a limitation in its long-standing single-controller model for scaling stateless workloads. To reuse idle capacity during regional failover, the platform needed multiple orchestrators to influence replica counts without overloading UDC, which remained on the normal deployment lifecycle path. The solution introduced the ServiceScale CRD and Service Scale Controller, allowing separate actors to record steady-state and temporary failover intent in Kubernetes and letting SSC reconcile the combined decision. Production work then exposed risks from informer caches that lagged reality and near-simultaneous multi-writer updates, including ReplicaSet metadata-spec drift that disrupted proportional scaling and could leave workloads stuck. Uber added fleet-wide detection, an automated UDC healer, a long-term scaling-path fix, and extensive kind integration tests, staging, and canaries; the year-long rollout avoided customer-impacting outages while establishing a more extensible control plane.
Ceilometer is Uber’s adaptive benchmarking framework for qualifying new server types, cloud SKUs, kernel upgrades, configuration changes, and machine-learning frameworks before production rollout. It replaces manual, fragmented testing with standardized configurations, automated cluster bring-up and service deployment, portable containerized harnesses, and reporting that includes server specifications, reference SLAs, normalization, resource utilization, cost efficiency, and reliability signals. Its distributed architecture runs synthetic and production-like workloads across dedicated machines, processes and enriches raw results, stores them in blob storage and a centralized data warehouse, and exposes them through an analytics service. The framework covers CPU and memory benchmarks, tools such as SpecCPU2017, SPECjbb2015, NetPerf, and FIO, plus stateful, stateless, batch, and ML workloads; Uber reports uses including regression detection, database validation, and canary testing of external upgrades.
ZGateway is a stateless proxy tier introduced to unify traffic from ZippyDB’s large, diverse client fleet and address the reliability limits of direct access. By terminating TLS, authorizing requests, applying admission control and shaping, and routing through regional tiers, it replaces a many-to-many connection mesh with bounded client-to-gateway and gateway-to-server hops. Its embedded thick C++ client batches and coalesces shard requests, while a shared pipeline supports pure-proxy and read-through-cache tiers; per-use-case metrics, traces, and quota usage provide operational visibility. The gateway handles more than 1 billion operations per second, carries about 40% of ZippyDB traffic, and adds about 6% computational overhead to an average use case, with traffic projected to exceed 60%; production work also consolidated transaction state handling across 100% of transaction traffic without a reliability regression.
NVIDIA Personal AI Router (PAIR) addresses the GPU bottleneck created when multi-agent workflows and concurrent local sessions generate many independent inference requests. It acts as a virtual inference router rather than an inference engine, proxying compatible Ollama and LM Studio interfaces, discovering and securely pairing local systems, and scheduling each request onto one eligible node based on readiness, engine state, model availability, workload, and GPU utilization. The agent keeps one familiar connection, while PAIR streams responses back and exposes placement through Jobs and metrics views; it does not merge GPUs, pool VRAM, or split a single request across machines. In an unofficial Hermes five-subagent demonstration, a three-device cluster averaged 8 minutes 48 seconds versus 18 minutes on one RTX Spark laptop, with results explicitly dependent on configuration and workload.
M3DB’s sharded placement algorithm lets replicas spread across nodes subject primarily to isolation-group constraints, but large clusters experienced broad failure and maintenance impacts. The subclustered placement algorithm partitions nodes into fixed-size, self-contained groups, each owning a distinct shard slice while preserving replica separation across isolation groups. It requires equal instance weights, scaling in multiples of the subcluster size, a replica factor that cannot change through AddReplica, and instancesPerSubcluster divisible by RF. During scaling, a greedy donation strategy simulates removing each candidate shard, ranks the resulting per-node skew, and chooses the shard that leaves the donor most balanced, avoiding a second rebalancing move. The resulting model contains recovery within subclusters, enables parallel maintenance, and reduces arbitrary shard sharing from O(cluster) to O(subcluster), while the original algorithm remains suitable for smaller clusters or impractical scaling steps.
Mixture-of-Kittens (MoK) is an open-source production mixture-of-experts (MoE) training megakernel designed for GB300 NVL72 systems, where the MoE layer had become a major training bottleneck and could consume more than half of end-to-end training time. It redesigns the layer around a single fully deterministic kernel that fuses communication and computation, minimizing CPU work and CPU-GPU synchronization while overlapping operations across the 72-GPU NVLink domain. The post describes target-specific scheduling, push-versus-pull communication choices, pipelined expert computation, ring token buffers, megakernel structure, determinism, and MXFP8 support for DeepSeek-V3-style layers. Across four model shapes, MoK is reported as up to 2.37x faster for MXFP8 forward and 1.78x for backward, while production training on 512 GPUs reached 1.41x tokens per second versus a DeepEP-based implementation. MoK is fully open-sourced and intended to remain modifiable for other platforms.
Schemaless triggers provides a scalable, fault-tolerant, and lossless way to listen for changes in Schemaless, Uber Engineering’s datastore using MySQL, and drives workflows including trip billing, analytics-warehouse ingestion, and cross-datacenter replication. Schemaless stores immutable cells and all their versions, allowing each shard to function as a partitioned change log with monotonically increasing added IDs while retaining random-access APIs. The framework tails shards through get_cells_for_shard, assigns them to worker processes under a leader, persists offsets in shared storage, and invokes column-specific callbacks such as bill_rider; handlers must be idempotent because retries can occur. Failed workers’ shards are reassigned, repeatedly failing cells can move to a separate queue, processing continues during leader or shared-storage outages although redistribution pauses, and replicas insulate triggers from database-node failures.
Schemaless is Uber's scalable, fault-tolerant datastore whose immutable cells are identified by row key, column name, and ref key, while JSON content remains schemaless and supports eventually consistent secondary indexes. Its worker nodes accept HTTP requests, route them to storage nodes, and aggregate results, while storage data is partitioned into typically 4096 shards, each replicated across one master and two minions in multiple data centers. Writes go to the master and replicate asynchronously, with idempotent retries, circuit breakers, failover, and configurable reads from masters or minions. To reduce loss during master failure, buffered writes persist each request to a primary and secondary cluster, though buffered data may not be immediately readable. Each shard maps to a MySQL database, where an auto-increment added_id primary key supports linear insertion and trigger processing, and MessagePack with ZLib compresses cell JSON blobs.
The best engineering reads, once a week.
