---
title: "Reliability"
description: "289 posts about Reliability, summarised, each linking to the original."
---

# Reliability
> 289 posts about Reliability, summarised, each linking to the original.

## Articles

### [Elixir at Ramp 🧪](https://yomu.fyi/post/elixir-at-ramp.md)
- Company: [Ramp](https://yomu.fyi/company/ramp.md)
- Author: Pablo Meier
- Published: May 24, 2021
- Type: Explainer

Ramp engineer Pablo describes why Elixir suited several critical services, while arguing that technology choice is only one part of engineering success. He contrasts BEAM, Erlang’s virtual machine, and OTP’s abstractions for reliable, concurrent systems with the adoption and developer-experience advantages of Elixir, including familiar syntax, Mix, iex, hot code reloading, doctests, macros, and sigils. Immutability and aliased module calls are presented as safeguards against hidden state changes and circular-import problems in growing codebases, though the post also notes that Erlang and Elixir lag behind bolted-on type checkers. Ramp uses Elixir for its real-time Authorization engine and Risk analysis platform; the Authorization service is described as exceptionally stable and flexible for rapid feature development. The company’s largest backend remains Python, and the conclusion prioritizes sleep, trust, ownership, and psychological safety over any specific stack.


### [Our Journey to Continuous Delivery at Grab (Part 2)](https://yomu.fyi/post/our-journey-to-continuous-delivery-at-grab-part-2.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Sylvain Bougerel
- Published: May 10, 2021
- Type: Problem & solution

Conveyor, an in-house continuous delivery system at Grab, introduces hermetic deployments by tracking application code alongside static and dynamic configuration parameters. This hermeticity guarantees that production releases use combinations of versioned artifacts previously verified in staging, preventing rollback incompatibilities. Conveyor replaces single multi-environment pipelines with decoupled pipelines, while automating cluster locking, deployment slot scheduling, release note generation, and canary monitoring with automated rollbacks. These automations reduced production deployment failure rates from 1.5% to an average of 0.3% over a three-month period. Grab also doubled the volume of production changes between 2018 and 2020 while saving more than 5,000 man-days of engineering effort in 2020.


### [Dynamic ProxySQL Query Rules](https://yomu.fyi/post/dynamic-proxysql-query-rules.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Apr 20, 2021
- Type: Problem & solution

Shopify uses ProxySQL to manage application connections to MySQL databases, but applying query rules across thousands of ProxySQL instances is difficult, especially during incidents. ProxySQL rules can reroute, rewrite, or reject regex-matched queries, yet incorrect patterns may cause unintended rejection, rewriting, data corruption, or downtime. To make rule changes safer, Shopify added dry-run support to a ProxySQL fork, using the mysql\_query\_rules table’s active and log fields; dry-run rules log matched queries without executing their actions. A sidecar polls a global key-value store, synchronizes dynamic rules to each ProxySQL instance, caches rules, supports optional TTLs, and uses exponential backoff for broad logging patterns. The system favors consistency during synchronization failures, while static startup rules cover requirements for full availability.


### [Customer Support Workforce Routing](https://yomu.fyi/post/customer-support-workforce-routing.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Suman Anand
- Published: Feb 5, 2021
- Type: Problem & solution

Grab replaced its third-party customer support routing software with an in-house workforce routing system for Livechat to gain better priority controls, bespoke configurations, and deeper analytics. The platform separates requests into distinct priority and business queues, using parallel workers that spend varied time slices dequeuing higher-priority issues like safety concerns. To prevent request starvation, workers operate out of sync across queue priority levels while dynamic queue limits cap incoming volume based on agent availability and performance. The system routes requests through an intermediate Agent Group layer, calculating eligibility scores from proficiency and concurrency metrics while managing per-agent locks to prevent over-allocation.


### [Graceful VM exits, some dials](https://yomu.fyi/post/graceful-vm-exits-some-dials.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Dec 29, 2020
- Type: Announcement

Fly.io introduces configurable VM shutdown behavior for applications whose clients, connections, or transactions need more than the default five-second termination window. After sending SIGINT, Fly normally waits five seconds before forcefully terminating a VM; the new kill\_timeout setting extends that wait to as much as five minutes on Shared CPU VMs or 24 hours on dedicated CPU VMs, while kill\_signal selects among several signals. The post demonstrates HAProxy’s soft-stop mode with SIGUSR1 to drain existing TCP connections and Postgres’s smart shutdown with SIGTERM to let open transactions commit before termination. It also describes infrastructure considerations: deploys and other releases can disrupt in-flight connections, so Fly drains connections where possible and offers ways to reduce reliance on shared HTTP and TLS handlers, plus stateless UDP and experimental stateless TCP load balancing.


### [Pharos - Searching Nearby Drivers on Road Network at Scale](https://yomu.fyi/post/pharos-searching-nearby-drivers-on-road-network-at-scale.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Hao Wu
- Published: Dec 22, 2020
- Type: Problem & solution

Ride-hailing allocation requires identifying nearby drivers based on actual routing distance and estimated time of arrival rather than straight-line haversine distance. Grab developed Pharos, a distributed in-memory spatial microservice that evaluates road networks to match fast-moving drivers with pick-up requests. The system partitions OpenStreetMap road graphs by city and vehicle vertical while indexing driver positions and edge-based nodes using Adaptive Radix Trees. To locate candidates, Pharos projects driver coordinates onto road segments as phantom nodes and runs Incremental Network Expansion to compute isochrone reaches. Read-write concurrency is managed by switching tree root snapshots, allowing concurrent reads while updates produce new roots for subsequent lookups.


### [Resiliency Planning for High-Traffic Events](https://yomu.fyi/post/resiliency-planning-for-high-traffic-events.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 10, 2020
- Type: Explainer

Shopify’s preparation for Black Friday Cyber Monday (BFCM) centers on building and maintaining resiliency practices for a high-traffic platform and large development teams. Regular load and stress tests verify required capacity, probe limits, detect regressions, and compare behavior across differently configured shops. Teams document expected user experiences and dependency failure modes in a resiliency matrix, then use game days to test that model against reality, including alerts, paging, operator readiness, and response procedures. Incident analysis treats disruptions as learning opportunities rather than stopping at a single “root cause”; nearer to BFCM, Shopify slows feature and service changes while increasing work on performance, reliability, scale, runbooks, tools, and bottlenecks. The source says these practices support a platform that powered over $5.1 billion in sales.


### [How to Reliably Scale Your Data Platform for High Volumes](https://yomu.fyi/post/how-to-reliably-scale-your-data-platform-for-high-volumes.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 8, 2020
- Type: Problem & solution

Shopify’s data platform prepared for BFCM 2020, when merchant sales exceeded $5.1 billion and platform throughput rose by an average of 150 percent. The Data Platform Engineering team focused on throughput risk, which grows with volume and is greatest near ingestion and processing, while prioritizing work through four service tiers. For Tier 1 and Tier 2 systems, engineers defined each service’s primary objective, identified adjustable controls such as job frequency, overlap intervals, CPU and memory requests, BigTable node pools, and ProxySQL connection queues, then used controlled load tests to expose capacity and recovery risks. The resulting work included cluster scaling, load-balancing adjustments, query and job tuning, and improved indices, schemas, dashboards, playbooks, SLOs, incident handling, and alerts. The post reports that BFCM 2020 completed without merchant-impacting incidents; only a handful of minor, self-healing incidents occurred, while monitoring and occasional tuning preserved spare capacity.


### [Organizing 2000 Developers for BFCM in a Remote World](https://yomu.fyi/post/organizing-2000-developers-for-bfcm-in-a-remote-world.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 7, 2020
- Type: Problem & solution

Shopify organized a cross-company program to prepare its platform for Black Friday and Cyber Monday (BFCM), coordinating more than 2,500 people responsible for over 1,000 deploys across services. Engineering Program Managers absorbed administrative, communication, and reporting work so BFCM leads could focus on technical challenges, while departmental champions assigned preparation tasks. Preparation covered maintenance, game days, load tests, capacity increases, weekly tracking, and a Caution Tape bot requiring risk-to-value assessments for pull requests in critical areas. Deploys were slowed through smaller batches, longer canaries, and a planned weeklong lock while communication used Slack, GitHub, documentation, presentations, emails, and office hours. The effort coincided with $5.1+ billion in BFCM sales and load testing plus resource scaling paid off, but developer productivity declined amid inconsistent messaging; future plans called for a more balanced approach, automation, and reusable communication templates.


### [Democratising Fare Storage at Scale Using Event Sourcing](https://yomu.fyi/post/democratising-fare-storage-at-scale-using-event-sourcing.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Sourabh Suman
- Published: Nov 23, 2020
- Type: Problem & solution

Grab's legacy system stored booking and fare details in a single relational table, creating a bloated booking entity that tracked only the latest fare state and hindered rapid feature iteration. To resolve scalability, stability, and debugging challenges across millions of daily bookings, the team developed Fare Storage using the Event Sourcing pattern. The new architecture persists all fare modification events chronologically in DynamoDB, backed by a cache for eventually consistent reads and message streaming for downstream processing. The platform employs optimistic locking with versioning to manage concurrent updates, enforces idempotency through client-generated transaction UUIDs, and delegates metadata serialization to an SDK to prevent storage API changes.


### [Uncovering the Truth Behind Lua and Redis Data Consistency](https://yomu.fyi/post/uncovering-the-truth-behind-lua-and-redis-data-consistency.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Allen Wang
- Published: Sep 7, 2020
- Type: Incident / postmortem

Grab experienced replica CPU usage spikes following service deployments in their master/replica Redis cluster, which caused failovers to spike to 100% CPU. Investigation revealed that a post-deployment Lua monitor script executed separately on both nodes and relied on non-deterministic HGETALL key ordering. Redis encodes hash objects as either ziplists or hashtables, and restoring from an RDB snapshot initializes small hashes as ziplists even if the master previously converted them to hashtables. This encoding discrepancy caused key ordering to diverge, preventing secondary data from deleting correctly and bloating dataset sizes. Grab resolved the issue by sorting the outputs of HKEYS and HGETALL within the Lua script to guarantee deterministic execution across nodes.


### [Building Reliable Mobile Applications](https://yomu.fyi/post/building-reliable-mobile-applications.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 14, 2020
- Type: Explainer

Shopify’s Point Of Sale (POS) app is mission-critical for brick-and-mortar merchants, because downtime can create long lineups, unhappy customers, and lost sales. Unlike server-side systems, mobile fixes depend on Apple and Google review, and merchants may delay installing approved updates, so the team focuses on preventing bugs and shortening recovery time. Its practices include layered automated and manual testing, component ownership with automatic Code Owners reviews, automated mobile “tophatting,” and weekly releases supported by a rotating Release Captain and ShipIt Mobile. Staged rollouts, beta flags, minimum-version enforcement for rare critical security fixes, and dedicated Primary and Secondary ATCs limit exposure and clarify incident response. ChatOps-based incident channels capture findings for root cause analysis, while rotating on-call responsibilities reduce context switching; these measures helped the app scale to larger merchants while shipping faster and improving reliability.


### [An Introduction to DNS Traffic Management](https://yomu.fyi/post/an-introduction-to-dns-traffic-management.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 5, 2020
- Type: Explainer

DNS traffic management applies data-driven DNS responses to direct clients to different service endpoints, extending DNS’s basic role of mapping domain names to IP addresses. The post introduces DNS through its history from HOSTS.TXT, explains zones, records, nameservers, root servers, TTL, and negative TTL, then connects caching behavior to routing decisions. Low TTLs can make changes visible sooner, but repeated DNS resolution adds latency: with a one-second resolution time, a 15-second TTL could add 5,760 seconds per day, compared with 1,440 seconds at 60 seconds; resolver behavior can also make delays slightly longer. Use cases include active-passive failover, active-active traffic sharing, progressive green-blue deployments, and location-based routing, while Shopify’s system evolved into an automated, self-served platform serving more than 40 domains across 12 teams and over 100M requests per 24 hours.


### [Using DNS Traffic Management to Add Resiliency to Shopify’s Services](https://yomu.fyi/post/using-dns-traffic-management-to-add-resiliency-to-shopify-s-services.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 5, 2020
- Type: Problem & solution

Shopify consolidated several team-specific DNS traffic-management approaches that relied on manual changes, pull requests, or provider APIs and had uneven failover, redundancy, and active/active capabilities. The replacement was a fully automated, self-served system for more than 40 domains owned by more than 12 teams and handling more than 100M requests per day. It stores domain configuration in Git, deploys it to two DNS providers, and exposes spy commands for fast updates and failovers. Its traffic-management design uses regionalized matching, endpoint-status filtering, weighted sharing, and fallback layers, while globally matching endpoints ensure an existing domain receives an answer. A later provider migration demonstrated the value of centralized ownership: the team updated Terraform and tooling, moved nameservers incrementally, and kept the change transparent to application owners despite concurrent incidents.


### [Continuous PostgreSQL Backups using WAL-G](https://yomu.fyi/post/continuous-postgresql-backups-using-wal-g.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Angelico de los Reyes
- Published: Aug 2, 2020
- Type: Tutorial

Continuous PostgreSQL backups require coordinating physical base snapshots with ongoing write-ahead log archiving to support point-in-time recovery. The open-source WAL-G utility simplifies this workflow by managing backup pushes and fetches to cloud storage services such as Amazon S3. In a typical setup on Ubuntu with PostgreSQL 12, envdir supplies AWS credentials and storage prefixes to WAL-G commands embedded directly in the PostgreSQL archive configuration. Regular base backups can be automated via cron jobs, which minimizes the volume of WAL archives that must be replayed during a restore. When recovering to a new instance, administrators fetch the latest base backup, define a restore command with targeted recovery timestamps in the configuration, and initiate replay using a recovery signal file.


### [Physical vs Logical Backups in PostgreSQL](https://yomu.fyi/post/physical-vs-logical-backups-in-postgresql.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Angelico de los Reyes
- Published: Jul 7, 2020
- Type: Explainer

PostgreSQL backups fall into two distinct categories: logical backups, which convert data into SQL command files, and physical backups, which copy the underlying file system storage. Logical backups generated with pg\_dump or pg\_dumpall allow single-database targeting and provide the only practical route for migrating across major Postgres versions with differing internal storage formats. Conversely, physical backups suit larger databases where long-running logical exports can degrade concurrent query performance and risk failure. Physical backups also integrate with Write Ahead Log files through tools like WAL-G to support Point in Time Recovery and minimize Recovery Point Objectives. Selecting between these two strategies depends on whether administrative simplicity, version portability, cluster size, or precise disaster recovery takes precedence.


### [Your Circuit Breaker is Misconfigured](https://yomu.fyi/post/your-circuit-breaker-is-misconfigured.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Feb 18, 2020
- Type: Explainer

Misconfigured circuit breakers can leave an application effectively unavailable during a dependency outage by allowing timed-out requests to consume worker capacity. Using Shopify’s Semian implementation, the post explains how name, error\_threshold, error\_timeout, half\_open\_resource\_timeout, and success\_threshold govern circuit opening, recovery, and wasted utilization across failing service instances. It models timeout spikes for a single worker, notes that separate names isolate services and instances, and presents an equation for steady-state additional utilization, while noting that context-switch costs are not included. A live test closely matched the equation’s prediction. In a Rails worker with two threads and 42 Redis instances, changing half\_open\_resource\_timeout from 0.25 seconds to 50ms and error\_timeout from 2 to 30 seconds reduced modeled extra utilization from 263% to 4%, with slower recovery; an author’s edit says success\_threshold does not affect steady-state utilization.


### [Developing Zoom Marketplace Apps w/ ngrok](https://yomu.fyi/post/developing-zoom-marketplace-apps-w-ngrok.md)
- Company: [Zoom](https://yomu.fyi/company/zoom.md)
- Author: Tim Slagle
- Published: Feb 14, 2020
- Type: Problem & solution

Developers frequently use ngrok to establish fast, introspectable tunnels to localhost when building Zoom Marketplace applications instead of configuring complex reverse proxies like NGINX or Apache. However, using basic ngrok tunnels in production exposes systems to short URL expiration windows, single points of failure, and scalability bottlenecks. To maintain application availability, developers can purchase an ngrok license to secure a service-level agreement and support. Additionally, teams should reserve dedicated subdomains rather than using auto-generated endpoints, run ngrok as a monitored background service on a cloud provider or data center, and place a load balancer in front of the tunnel to handle heavy traffic.


### [How We Prevented App Performance Degradation from Sudden Ride Demand Spikes](https://yomu.fyi/post/how-we-prevented-app-performance-degradation-from-sudden-ride-demand-s.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Corey Scott
- Published: Jan 8, 2020
- Type: Problem & solution

Grab experienced severe system strain when sudden localized spikes in ride demand, triggered by events like heavy rain or concert dismissals, coincided with driver shortages. These localized bursts overloaded the platform and degraded the experience for users outside the affected areas. To mitigate this, engineers created the Spampede filter, a circuit-breaker mechanism placed at the start of the booking pipeline. The filter converts pickup locations into Geohash Integer buckets and partitions time using Unix timestamps, tracking unfulfilled requests in Redis with atomic increments and time-to-live expirations. When unallocated requests exceed configured thresholds within a specific bucket, the system immediately short-circuits new incoming bookings to protect overall platform stability.


### [Let It Crash: Best Practices for Handling Node.js Errors on Shutdown](https://yomu.fyi/post/let-it-crash-best-practices-for-handling-node-js-errors-on-shutdown.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Julián Duque
- Published: Dec 18, 2019
- Type: Tutorial

Node.js applications can lose visibility into failures and experience downtime when processes crash or shut down without useful logging and recovery practices. The post presents lessons from consulting and a talk on the Node.js process lifecycle, focusing on graceful shutdown, error visibility, and fast restart after catastrophic errors. It recommends keeping startup lean by prebuilding data or compiling assets, then contrasts process.exit(), which accepts an exit code, with process.abort(), which terminates immediately and may generate a core dump for llnode-based postmortem debugging. It also explains beforeExit and exit, including their differing asynchronous-work behavior, and introduces SIGTERM and SIGINT as operating-system signals that applications can handle. The intended result is a restartable process strategy that reduces downtime, while preserving evidence needed to investigate crashes.


[Newer posts](https://yomu.fyi/topic/reliability/page/12.md) · [Older posts](https://yomu.fyi/topic/reliability/page/14.md)