Copy-and-Replay vs Branch-at-a-Point: Two Backup Recovery Paths
A side-by-side comparison of copy-and-replay recovery and branch-at-a-point restores, showing why restore time scales with data size in one model and not the other.
Loading…
Topic
Reliability
289 posts about Reliability. Every summary links to the original.
- 289 posts
- 46 companies
- 2 explainers
See the idea
Reliability explainers
The SQS Decision Most Diagrams Skip: Normal Work vs. Broken Work
Amazon SQS connects producers and workers, but its real design test is failure. Compare the processing path with the signals and controls needed when queue operations break.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Safely Adding NOT NULL Columns to Your Database Tables
Shopify’s Database Migrations team investigated whether Large Hadron Migrator (LHM) can safely add NOT NULL columns to existing MySQL tables with minimal downtime. LHM creates a shadow table, installs triggers for INSERT, UPDATE, and DELETE operations, and copies records in batches before renaming the tables. The experiments varied whether the new column had a DEFAULT value or UNIQUE index and whether MySQL used strict or non-strict mode, evaluating trigger compatibility and record counts. A NOT NULL column with a DEFAULT was backward compatible, but adding a UNIQUE index could cause data loss; without a DEFAULT, strict mode broke compatibility while non-strict mode used implicit defaults, which may be undesirable.
Six Tips for Staying Technical as a CTO
The article examines how a CTO can remain technically credible after moving from the individual contributor track into leadership. It argues that leadership inevitably reduces technical depth because performance reviews, one-on-ones, organizational communication, and other people responsibilities limit time in code, making the transition an intentional choice for those who value managing people. Its recommended practices include voracious reading, coding as a hobby, participating in hackathons, learning directly from more technical colleagues, maintaining a private network of technical advisors, and aggressively clearing bugs, incident-remediation work, and backlog. The author illustrates hands-on learning with Shopify’s Spin environment, which provisions a cloud development instance and opens a repository in about 30 seconds, and concludes that controlling operational issues preserves time for innovation and technical leadership.
10 Tips for Building Resilient Payment Systems
The post distills lessons from five years working on Shopify’s payment infrastructure into guidance for building systems that tolerate failure and large-scale traffic. It frames the advice partly around onboarding developers who were new to payments or Shopify’s scale, and notes that load testing for BFCM 2021 began in July with globally distributed load generation. It recommends low timeouts across HTTP clients and data stores, circuit breakers such as Semian, and capacity reasoning with Little’s Law, rate limiting, and load shedding. Monitoring should cover latency, traffic, errors, and saturation, while structured, machine-readable logs make centralized records searchable; load tests use benchmark gateways that mimic production partner latency and capacity. For incidents, Shopify uses defined roles, Slack-based coordination, and retrospectives that turn incorrect assumptions into safeguards and training material.
Spin Infrastructure Adventures: Containers, Systemd, and CGroups
Spin’s infrastructure team investigated instance instability after Kubernetes nodes experienced frequent failures, including roughly five nodes per day, and instances showed memory use that could exceed their assigned limits. Spin runs instances as Kubernetes pods, with systemd inside each container managing initialization and processes, and the team compared resource behavior across kubectl exec and Spin shell sessions. Stress tests showed that processes started through kubectl exec entered the kubepods cgroup hierarchy, while processes launched through Spin shell entered a hierarchy with an effectively unlimited memory limit, causing usage to appear on the node rather than the pod. Experiments comparing containerd, Docker, and Podman identified systemd cgroup delegation as the relevant difference: Podman isolated systemd and its child processes correctly, while the production setup used cgroupfs. The team identified switching COS to the systemd cgroup driver as the forthcoming fix, while noting that the bind-mounted cgroup filesystem enabled the escape.
The Story Behind Shopify’s Isospin Tooling
Shopify’s Isospin is the systemd-based core of Spin, its in-house cloud development platform, designed to run multiple applications inside a single Linux VM. The earlier implementation relied on shell scripts, but growing application counts made it difficult to split applications into components, express dependencies, schedule jobs, and isolate processes. Isospin uses systemd unit graphs, targets, template units, and generators to create service configurations dynamically, including per-application copies of dependencies such as MySQL and Elasticsearch. Additional tooling assigns semi-stable ports by hashing service names and uses systemd readiness notifications, including a notify-port wrapper that checks HTTP availability; the resulting system reuses systemd’s dependency and health mechanisms, with more complex cross-application relationships under investigation.
Navigating Recurring Payments in India: A Backend Perspective
India’s October 2021 Reserve Bank of India (RBI) directive required recurring payments to use payer-authorized e-mandates, Additional Factor of Authentication (AFA), and pre-debit notifications at least 24 hours before charges. Shopify’s billing platform therefore had to model e-mandates, support recurring and one-time payments across cards and Unified Payments Interface (UPI), and integrate a local payment provider. The payment service reused its source object for the e-mandate token, recorded the temporary authorization as a charge for auditability, and refunded it asynchronously. To avoid two consecutive card authentication flows, the team used recurring-payment tokenization for both payment paths, accepting an atypical notification experience as a short-term trade-off; UPI required asynchronous webhook-driven handling and a hard mandate limit. After the beta launch, billing-payment success rates in India increased from pre-project levels, and the team said it would soon make UPI generally available to Indian merchants.
Making Open Source Safer for Everyone with Shopify’s Bug Bounty Program
Shopify describes how a bug bounty report led to application fixes and Rails contributions aimed at making open-source software safer. The 2021 Shopify Experiments report showed that a leaked development-environment secret could be used to generate trusted-looking data and reach remote code execution in a local development environment. The root cause was ActiveSupport’s MessageEncryptor using Ruby’s unsafe Marshal serializer by default in Rails versions up to 7.0.X, allowing untrusted data to trigger deserialization risks. Shopify first configured JSON in its application, then helped change the default serializers for MessageEncryptor and MessageVerifier to JSON in Rails 7.1.X, with an upgrade path for existing applications. It also developed a patch that monitors Marshal load and dump calls, raises on unexpected gem-originated calls, and runs in local development and continuous integration to prevent new usage.
Logbook - 2022-05-13
The May 13, 2022 log records Fly.io changes and fixes from the preceding week, spanning infrastructure capacity, account self-service, API maintenance, documentation, and customer-facing behavior. New servers in syd, iad, and dfw, plus three workers in ord, addressed capacity shortages that had caused provisioning or deployment problems. The API moved to Ruby 3.1, with anticipated performance and security-patch benefits, while an internal Elixir UI guide and clearer app and machine name validation errors aimed to reduce development and debugging friction. Operational work permanently disabled consul-templaterb on fly-proxy nodes and fixed a fly-proxy memory leak, giving those nodes more CPU/RAM headroom; an MTU workaround was intended to improve IPv6 tunnel connectivity, while a larger fix remains. Other fixes covered account-deactivation guidance, personalized support email addresses for paid plans, billing, volume metrics, forum registration, and a minor “X daysago” display error.
Maestro: The Orchestration Language Powering Shopify Flow
Shopify developed Maestro, a domain-specific orchestration language, to decouple Shopify Flow’s merchant-facing visual editor from its execution engine, whose requirements differ in declarative usability versus scalable, fault-tolerant execution. The team introduced a horizontally scalable Maestro engine, translated existing Flow workflows into Maestro, migrated them incrementally, and later connected a new visual language directly to Maestro; all workflows had moved to the new editor and engine by early 2022. Maestro coordinates calls to host-language functions rather than implementing service or database operations, while durable checkpoints capture completed work and let the interpreter recover after crashes with at-least-once semantics for primitive calls. During activation, static usage analysis identifies needed returned attributes for possible query optimization, and compilation removes runtime-irrelevant definitions before deployment; checkpoints also support execution monitoring, while planned work includes richer language features, orchestration merging, and workflow preview and testing.
Möbius: Shopify’s Unified Edge
Shopify had more than six traffic paths for its core, services, PII-restricted systems, and mTLS-authenticated APIs, making platform-wide features harder to deploy and traffic behavior harder to observe. Möbius and Shopify Front End (SFE) were designed together: Möbius terminates TCP and TLS near clients and forwards requests to SFE, while SFE routes them from regional clusters to application clusters. SFE uses nginx deployments with custom Lua and a centralized API fed by Kubernetes controllers, enabling routing and active/active or active/passive load balancing without DNS TTL delays. Möbius automates edge domain configuration, DNS ownership validation, and certificate acquisition through vendor APIs, so application owners generally only need to change DNS. The unified design standardizes edge protection and observability, supports future edge capabilities, and leaves exceptions for services whose dependencies could create crash loops.
Code Ranges: A Deeper Look at Ruby Strings
Ruby's string model supports mutable and immutable operations, per-object or per-file freezing, and more than 100 encodings, while permitting byte sequences invalid for their attached encoding. This flexibility helps legacy and unusual platforms but makes encoding checks, validity scans, and character-boundary handling part of many operations. To reduce repeated O(n) validation scans, Ruby implementations cache each string's state in a four-valued code range: unknown, 7bit, valid, or broken. The post explains how these values map to public methods, how MRI stores them in two header bits, and how JRuby and TruffleRuby represent them differently. It also describes code ranges as inputs to fast and slow paths and as useful but fallible cache information, especially for native extensions and string-heavy metaprogramming.
7 Tips For Optimizing Apache Flink Applications
Shopify presents lessons from operating large, stateful Apache Flink applications on Kubernetes with Google Kubernetes Engine, RocksDB, and checkpoints and savepoints stored in Google Cloud Storage. The guidance begins with profiling tools, including async-profiler, VisualVM, jemalloc with jeprof, and Eclipse Memory Analyzer, to investigate CPU, heap, native-memory, and leak-related problems. Disabling Kryo fallback exposed unsupported serializers such as Scala BigDecimal and Scala algebraic data types; replacing them with supported alternatives increased throughput by 20 percent. The post also recommends separate configuration profiles for high-throughput backfills and low-latency steady state, while describing a RocksDB native-memory failure in which disabling the block cache stopped repeated out-of-memory crashes without affecting application performance.
That Old Certificate Expired and Started an Outage. This is What Happened Next
On September 30th, 2021, Shopify nearly suffered a platform-wide outage after Let’s Encrypt’s old root certificate expired, exposing stale certificate data in internal clients rather than at the public edge. At 10:15 a.m., monitoring detected certificate errors after deployment-related connection changes, and investigation traced them to Ruby services that depended directly or indirectly on httpclient, whose embedded cacert.pem was six years old and did not trust the applicable certificate chain. Shopify forked the library, replaced cacert.pem with a current root certificate bundle, deployed it to canaries, and automated pull requests for affected repositories. The temporary fix restored the monolith, while systems were added to track root-certificate changes and update the fork automatically. Deployment locking kept most monolith pods from cycling, limiting the incident to canaries and causing secondary effects including delayed order confirmations, throttled marketing campaigns, and missing product recommendations.
Upgrading MySQL at Shopify
In September 2021, Shopify completed its cutover from Percona Server 5.7.21 to 5.7.32 across a MySQL fleet spanning hundreds of horizontally sharded databases and thousands of virtual machines. The upgrade targeted replication improvements from transaction writesets, along with security and stability fixes, while avoiding the larger risks of a major-version change. The Database Platform team extended Mason into a declarative topology system and used Chef roles to provision upgraded VMs, restore backups, run mysql_upgrade when required, configure replication, and populate replacement topologies. Extensive rollback testing exposed a critical downgrade bug; after patching 5.7.21, Shopify proceeded in staggered batches beginning August 25 and completed the final shards on September 7, while documenting a repeatable process intended to make future upgrades faster and safer.
Shopify's Path to a Faster Trino Query Execution: Custom Verification, Benchmarking, and Profiling Tooling
Shopify needed safer ways to evaluate Trino configuration changes and software upgrades as its forked, hundreds-node cluster handled more data and data scientists depended on fast queries. The Data Reliability team built a lightweight Python library that centralizes Trino query orchestration, then exposed it through PyTest fixtures for repeatable verification, including correctness checks and candidate-versus-control comparisons. The broader framework also supports benchmarking and profiling concepts such as replaying historical queries, reproducing traffic patterns, and testing autoscaling, although the project ultimately focused most of its performance-testing scope on verification. Extensive correctness tests helped the Data Foundations team complete a successful Trino upgrade, while p95 query execution time remained stable over the reported period; the library was designed to leave room for more advanced interfaces later.
Debugging Systems in the Cloud: MySQL, Kubernetes, and Cgroups
KateSQL, Shopify’s Database-as-a-Service platform on GKE, investigated why otherwise comparable MySQL Pods took between 10 and 30 minutes to be created during instance replacement. Measurements found extreme differences for Pods with identical resources and datasets: one required 2,120 seconds for initialization and 1,104 seconds for startup, while another took 74 and 17 seconds, respectively; affected Pods also accumulated more slow queries involving temporary memory tables. Profiling showed InnoDB’s single-threaded buffer pool initialization spending substantial time in mmap calls and the Linux memory cgroup function mem_cgroup_commit_charge, while affected nodes had roughly 50,000 memory cgroups. The team linked the slowdown to a Linux kernel cgroup memory leak and slab reparenting behavior, then improved performance through node replacement, memory.force_empty testing, and GKE 1.18 upgrades with relevant fixes.
Shard Balancing: Moving Shops Confidently with Zero-Downtime at Terabyte-scale
Shopify’s MySQL shards can become unevenly utilized as merchants grow after being assigned to arbitrary shards, increasing over-utilization risk while leaving other capacity underused. The post describes a strategy for selecting shop placements using historical utilization and traffic data, simulations, and potentially other metrics such as shop size, GMV, and move time. It then explains online shop migration with Ghostferry, which batch-copies records and tails MySQL’s binary log before a brief cutover updates the routing table and directs new writes to the target pod. Verification checks data completeness, integrity, and stale traffic before old data is pruned; the migration system’s central algorithm is also represented in a TLA+ formal specification. The stated outcome is a more balanced infrastructure that can isolate high-traffic merchants across pods with virtually zero consumer-facing downtime.
5 Steps for Building Machine Learning Models for Business
The article presents a five-step workflow for deciding when and how to build machine learning products in a business setting. It recommends delaying machine learning for new products, establishing data-collection foundations, and testing simple heuristics as baselines before investing in a model. For an initial model, teams should favor simple algorithms, limited feature sets, and off-the-shelf solutions where appropriate, then measure business-aligned impact before optimizing. The workflow also calls for planned model iteration and explicit comparison with existing versions, while accounting for prediction stability when changes could affect user trust. The authors say they used these practices at Shopify to help build and scale machine learning products and to productionize models intended to drive impact.
High Availability by Offloading Work Into the Background
Unpredictable traffic spikes, slow third-party payment requests, and resource-intensive image processing can overwhelm an application, making it slow or unavailable; the text presents background jobs as a way to preserve meaningful user interactions. High availability is defined not merely as server uptime, but as users being able to interact with the application when needed. The application server places a task message in a queue and continues serving requests, while workers process jobs asynchronously, with support for retries, parallelization, prioritization, and event- or time-based scheduling. Offloading improves response times and buffers spikes, but changes immediate guarantees: an upload may be accepted before processing succeeds, and errors may be handled later. For distributed systems, data-based messaging such as RabbitMQ can connect different services, while Kafka's append-only, replayable event log supports shared consumption and event sourcing.
Debugging High Latency Due to Context Leaks
Market-Store, Grab's feature store for real-time machine learning features, experienced latency spikes from under 200 milliseconds to 2 seconds as traffic grew. Metrics and logs showed no direct correlation to API issues, but heap profiling with PPROF revealed continuously increasing memory held by child contexts. Further analysis tracked the leak to an update in Grab's open-source Async Library, which switched background contexts to uncancelled task contexts for worker runners. Because parent contexts maintained references to these uncancelled child contexts, the garbage collector could not reclaim their memory. This progressive memory exhaustion directly degraded API latency.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.