Copy-and-Replay vs Branch-at-a-Point: Two Backup Recovery Paths
A side-by-side comparison of copy-and-replay recovery and branch-at-a-point restores, showing why restore time scales with data size in one model and not the other.
Loading…
Topic
Reliability
287 posts about Reliability. Every summary links to the original.
- 287 posts
- 45 companies
- 2 explainers
See the idea
Reliability explainers
The SQS Decision Most Diagrams Skip: Normal Work vs. Broken Work
Amazon SQS connects producers and workers, but its real design test is failure. Compare the processing path with the signals and controls needed when queue operations break.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Dynamic ProxySQL Query Rules
Shopify uses ProxySQL to manage application connections to MySQL databases, but applying query rules across thousands of ProxySQL instances is difficult, especially during incidents. ProxySQL rules can reroute, rewrite, or reject regex-matched queries, yet incorrect patterns may cause unintended rejection, rewriting, data corruption, or downtime. To make rule changes safer, Shopify added dry-run support to a ProxySQL fork, using the mysql_query_rules table’s active and log fields; dry-run rules log matched queries without executing their actions. A sidecar polls a global key-value store, synchronizes dynamic rules to each ProxySQL instance, caches rules, supports optional TTLs, and uses exponential backoff for broad logging patterns. The system favors consistency during synchronization failures, while static startup rules cover requirements for full availability.
Customer Support Workforce Routing
Grab replaced its third-party customer support routing software with an in-house workforce routing system for Livechat to gain better priority controls, bespoke configurations, and deeper analytics. The platform separates requests into distinct priority and business queues, using parallel workers that spend varied time slices dequeuing higher-priority issues like safety concerns. To prevent request starvation, workers operate out of sync across queue priority levels while dynamic queue limits cap incoming volume based on agent availability and performance. The system routes requests through an intermediate Agent Group layer, calculating eligibility scores from proficiency and concurrency metrics while managing per-agent locks to prevent over-allocation.
Graceful VM exits, some dials
Fly.io introduces configurable VM shutdown behavior for applications whose clients, connections, or transactions need more than the default five-second termination window. After sending SIGINT, Fly normally waits five seconds before forcefully terminating a VM; the new kill_timeout setting extends that wait to as much as five minutes on Shared CPU VMs or 24 hours on dedicated CPU VMs, while kill_signal selects among several signals. The post demonstrates HAProxy’s soft-stop mode with SIGUSR1 to drain existing TCP connections and Postgres’s smart shutdown with SIGTERM to let open transactions commit before termination. It also describes infrastructure considerations: deploys and other releases can disrupt in-flight connections, so Fly drains connections where possible and offers ways to reduce reliance on shared HTTP and TLS handlers, plus stateless UDP and experimental stateless TCP load balancing.
Pharos - Searching Nearby Drivers on Road Network at Scale
Ride-hailing allocation requires identifying nearby drivers based on actual routing distance and estimated time of arrival rather than straight-line haversine distance. Grab developed Pharos, a distributed in-memory spatial microservice that evaluates road networks to match fast-moving drivers with pick-up requests. The system partitions OpenStreetMap road graphs by city and vehicle vertical while indexing driver positions and edge-based nodes using Adaptive Radix Trees. To locate candidates, Pharos projects driver coordinates onto road segments as phantom nodes and runs Incremental Network Expansion to compute isochrone reaches. Read-write concurrency is managed by switching tree root snapshots, allowing concurrent reads while updates produce new roots for subsequent lookups.
Resiliency Planning for High-Traffic Events
Shopify’s preparation for Black Friday Cyber Monday (BFCM) centers on building and maintaining resiliency practices for a high-traffic platform and large development teams. Regular load and stress tests verify required capacity, probe limits, detect regressions, and compare behavior across differently configured shops. Teams document expected user experiences and dependency failure modes in a resiliency matrix, then use game days to test that model against reality, including alerts, paging, operator readiness, and response procedures. Incident analysis treats disruptions as learning opportunities rather than stopping at a single “root cause”; nearer to BFCM, Shopify slows feature and service changes while increasing work on performance, reliability, scale, runbooks, tools, and bottlenecks. The source says these practices support a platform that powered over $5.1 billion in sales.
How to Reliably Scale Your Data Platform for High Volumes
Shopify’s data platform prepared for BFCM 2020, when merchant sales exceeded $5.1 billion and platform throughput rose by an average of 150 percent. The Data Platform Engineering team focused on throughput risk, which grows with volume and is greatest near ingestion and processing, while prioritizing work through four service tiers. For Tier 1 and Tier 2 systems, engineers defined each service’s primary objective, identified adjustable controls such as job frequency, overlap intervals, CPU and memory requests, BigTable node pools, and ProxySQL connection queues, then used controlled load tests to expose capacity and recovery risks. The resulting work included cluster scaling, load-balancing adjustments, query and job tuning, and improved indices, schemas, dashboards, playbooks, SLOs, incident handling, and alerts. The post reports that BFCM 2020 completed without merchant-impacting incidents; only a handful of minor, self-healing incidents occurred, while monitoring and occasional tuning preserved spare capacity.
Organizing 2000 Developers for BFCM in a Remote World
Shopify organized a cross-company program to prepare its platform for Black Friday and Cyber Monday (BFCM), coordinating more than 2,500 people responsible for over 1,000 deploys across services. Engineering Program Managers absorbed administrative, communication, and reporting work so BFCM leads could focus on technical challenges, while departmental champions assigned preparation tasks. Preparation covered maintenance, game days, load tests, capacity increases, weekly tracking, and a Caution Tape bot requiring risk-to-value assessments for pull requests in critical areas. Deploys were slowed through smaller batches, longer canaries, and a planned weeklong lock while communication used Slack, GitHub, documentation, presentations, emails, and office hours. The effort coincided with $5.1+ billion in BFCM sales and load testing plus resource scaling paid off, but developer productivity declined amid inconsistent messaging; future plans called for a more balanced approach, automation, and reusable communication templates.
Democratising Fare Storage at Scale Using Event Sourcing
Grab's legacy system stored booking and fare details in a single relational table, creating a bloated booking entity that tracked only the latest fare state and hindered rapid feature iteration. To resolve scalability, stability, and debugging challenges across millions of daily bookings, the team developed Fare Storage using the Event Sourcing pattern. The new architecture persists all fare modification events chronologically in DynamoDB, backed by a cache for eventually consistent reads and message streaming for downstream processing. The platform employs optimistic locking with versioning to manage concurrent updates, enforces idempotency through client-generated transaction UUIDs, and delegates metadata serialization to an SDK to prevent storage API changes.
Uncovering the Truth Behind Lua and Redis Data Consistency
Grab experienced replica CPU usage spikes following service deployments in their master/replica Redis cluster, which caused failovers to spike to 100% CPU. Investigation revealed that a post-deployment Lua monitor script executed separately on both nodes and relied on non-deterministic HGETALL key ordering. Redis encodes hash objects as either ziplists or hashtables, and restoring from an RDB snapshot initializes small hashes as ziplists even if the master previously converted them to hashtables. This encoding discrepancy caused key ordering to diverge, preventing secondary data from deleting correctly and bloating dataset sizes. Grab resolved the issue by sorting the outputs of HKEYS and HGETALL within the Lua script to guarantee deterministic execution across nodes.
Building Reliable Mobile Applications
Shopify’s Point Of Sale (POS) app is mission-critical for brick-and-mortar merchants, because downtime can create long lineups, unhappy customers, and lost sales. Unlike server-side systems, mobile fixes depend on Apple and Google review, and merchants may delay installing approved updates, so the team focuses on preventing bugs and shortening recovery time. Its practices include layered automated and manual testing, component ownership with automatic Code Owners reviews, automated mobile “tophatting,” and weekly releases supported by a rotating Release Captain and ShipIt Mobile. Staged rollouts, beta flags, minimum-version enforcement for rare critical security fixes, and dedicated Primary and Secondary ATCs limit exposure and clarify incident response. ChatOps-based incident channels capture findings for root cause analysis, while rotating on-call responsibilities reduce context switching; these measures helped the app scale to larger merchants while shipping faster and improving reliability.
An Introduction to DNS Traffic Management
DNS traffic management applies data-driven DNS responses to direct clients to different service endpoints, extending DNS’s basic role of mapping domain names to IP addresses. The post introduces DNS through its history from HOSTS.TXT, explains zones, records, nameservers, root servers, TTL, and negative TTL, then connects caching behavior to routing decisions. Low TTLs can make changes visible sooner, but repeated DNS resolution adds latency: with a one-second resolution time, a 15-second TTL could add 5,760 seconds per day, compared with 1,440 seconds at 60 seconds; resolver behavior can also make delays slightly longer. Use cases include active-passive failover, active-active traffic sharing, progressive green-blue deployments, and location-based routing, while Shopify’s system evolved into an automated, self-served platform serving more than 40 domains across 12 teams and over 100M requests per 24 hours.
Using DNS Traffic Management to Add Resiliency to Shopify’s Services
Shopify consolidated several team-specific DNS traffic-management approaches that relied on manual changes, pull requests, or provider APIs and had uneven failover, redundancy, and active/active capabilities. The replacement was a fully automated, self-served system for more than 40 domains owned by more than 12 teams and handling more than 100M requests per day. It stores domain configuration in Git, deploys it to two DNS providers, and exposes spy commands for fast updates and failovers. Its traffic-management design uses regionalized matching, endpoint-status filtering, weighted sharing, and fallback layers, while globally matching endpoints ensure an existing domain receives an answer. A later provider migration demonstrated the value of centralized ownership: the team updated Terraform and tooling, moved nameservers incrementally, and kept the change transparent to application owners despite concurrent incidents.
Continuous PostgreSQL Backups using WAL-G
Continuous PostgreSQL backups require coordinating physical base snapshots with ongoing write-ahead log archiving to support point-in-time recovery. The open-source WAL-G utility simplifies this workflow by managing backup pushes and fetches to cloud storage services such as Amazon S3. In a typical setup on Ubuntu with PostgreSQL 12, envdir supplies AWS credentials and storage prefixes to WAL-G commands embedded directly in the PostgreSQL archive configuration. Regular base backups can be automated via cron jobs, which minimizes the volume of WAL archives that must be replayed during a restore. When recovering to a new instance, administrators fetch the latest base backup, define a restore command with targeted recovery timestamps in the configuration, and initiate replay using a recovery signal file.
Physical vs Logical Backups in PostgreSQL
PostgreSQL backups fall into two distinct categories: logical backups, which convert data into SQL command files, and physical backups, which copy the underlying file system storage. Logical backups generated with pg_dump or pg_dumpall allow single-database targeting and provide the only practical route for migrating across major Postgres versions with differing internal storage formats. Conversely, physical backups suit larger databases where long-running logical exports can degrade concurrent query performance and risk failure. Physical backups also integrate with Write Ahead Log files through tools like WAL-G to support Point in Time Recovery and minimize Recovery Point Objectives. Selecting between these two strategies depends on whether administrative simplicity, version portability, cluster size, or precise disaster recovery takes precedence.
Your Circuit Breaker is Misconfigured
Misconfigured circuit breakers can leave an application effectively unavailable during a dependency outage by allowing timed-out requests to consume worker capacity. Using Shopify’s Semian implementation, the post explains how name, error_threshold, error_timeout, half_open_resource_timeout, and success_threshold govern circuit opening, recovery, and wasted utilization across failing service instances. It models timeout spikes for a single worker, notes that separate names isolate services and instances, and presents an equation for steady-state additional utilization, while noting that context-switch costs are not included. A live test closely matched the equation’s prediction. In a Rails worker with two threads and 42 Redis instances, changing half_open_resource_timeout from 0.25 seconds to 50ms and error_timeout from 2 to 30 seconds reduced modeled extra utilization from 263% to 4%, with slower recovery; an author’s edit says success_threshold does not affect steady-state utilization.
Developing Zoom Marketplace Apps w/ ngrok
Developers frequently use ngrok to establish fast, introspectable tunnels to localhost when building Zoom Marketplace applications instead of configuring complex reverse proxies like NGINX or Apache. However, using basic ngrok tunnels in production exposes systems to short URL expiration windows, single points of failure, and scalability bottlenecks. To maintain application availability, developers can purchase an ngrok license to secure a service-level agreement and support. Additionally, teams should reserve dedicated subdomains rather than using auto-generated endpoints, run ngrok as a monitored background service on a cloud provider or data center, and place a load balancer in front of the tunnel to handle heavy traffic.
How We Prevented App Performance Degradation from Sudden Ride Demand Spikes
Grab experienced severe system strain when sudden localized spikes in ride demand, triggered by events like heavy rain or concert dismissals, coincided with driver shortages. These localized bursts overloaded the platform and degraded the experience for users outside the affected areas. To mitigate this, engineers created the Spampede filter, a circuit-breaker mechanism placed at the start of the booking pipeline. The filter converts pickup locations into Geohash Integer buckets and partitions time using Unix timestamps, tracking unfulfilled requests in Redis with atomic increments and time-to-live expirations. When unallocated requests exceed configured thresholds within a specific bucket, the system immediately short-circuits new incoming bookings to protect overall platform stability.
Let It Crash: Best Practices for Handling Node.js Errors on Shutdown
Node.js applications can lose visibility into failures and experience downtime when processes crash or shut down without useful logging and recovery practices. The post presents lessons from consulting and a talk on the Node.js process lifecycle, focusing on graceful shutdown, error visibility, and fast restart after catastrophic errors. It recommends keeping startup lean by prebuilding data or compiling assets, then contrasts process.exit(), which accepts an exit code, with process.abort(), which terminates immediately and may generate a core dump for llnode-based postmortem debugging. It also explains beforeExit and exit, including their differing asynchronous-work behavior, and introduces SIGTERM and SIGINT as operating-system signals that applications can handle. The intended result is a restartable process strategy that reduces downtime, while preserving evidence needed to investigate crashes.
Four Steps to Creating Effective Game Day Tests
Game Day tests deliberately trigger failure modes in production systems and examine whether automated behavior and human response match expectations. Shopify frames the practice as preparation for outages that can arise despite code checks, reviews, and other safeguards. The method starts by inventorying possible failures, then selecting scenarios by impact and simulation difficulty rather than estimated likelihood. Each scenario becomes a controlled experiment with prerequisites, testable hypotheses, expected alerts and downstream behavior, plus recovery steps recorded in a spreadsheet. During execution, teams track results, communication, and work distribution; afterward they fix gaps, review coordination, update response guidelines, and rerun tests to verify changes, with periodic testing keeping systems and people prepared.
Building Resilient GraphQL APIs Using Idempotency
Shopify’s Payment Service uses API-level idempotency to make GraphQL payment mutations resilient to timeouts, connection failures, and ambiguous outcomes that could otherwise leave merchants unpaid or double-charge buyers. The design places a UUID-based idempotency key directly in each mutation input, identifies requests by client plus key, and locks concurrent duplicates so retries receive a 409 response while the original call is processing. An IncomingRequest record stores request state and completed recovery points, allowing completed responses to be replayed and incomplete attempts to recover before continuing. Handlers divide work into no-side-effect, transactional local-side-effect, and remote-side-effect steps, each optionally providing run and recover functions. The approach adds database-write overhead and requires careful compatibility testing when recovery points change, but the team reports that its structured handlers are clear and maintainable.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.