Copy-and-Replay vs Branch-at-a-Point: Two Backup Recovery Paths
A side-by-side comparison of copy-and-replay recovery and branch-at-a-point restores, showing why restore time scales with data size in one model and not the other.
Loading…
Topic
Reliability
287 posts about Reliability. Every summary links to the original.
- 287 posts
- 45 companies
- 2 explainers
See the idea
Reliability explainers
The SQS Decision Most Diagrams Skip: Normal Work vs. Broken Work
Amazon SQS connects producers and workers, but its real design test is failure. Compare the processing path with the signals and controls needed when queue operations break.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Championing CyberSecurity: Grab's bug bounty programme in 2023
Grab reviewed the performance and milestones of its security bug bounty programme for 2023, marking eight years of running the initiative. Operating quarterly campaigns through HackerOne alongside dedicated internal teams, the company processed approximately 1,000 vulnerability submissions over the year, including 400 reports during Q1 alone. Cumulative payouts to external security researchers surpassed $700,000 since the programme's 2015 launch. To adapt to an expanding service portfolio, Grab broadened its scope to encompass joint ventures and acquisitions while introducing diversified financial and recognition-based rewards.
Abstraction Engineering
Ramp’s early product encoded opinionated limits, but by fall 2022 customers needed configuration based on departments, accounting fields, subsidiaries, budgets, and HRIS fields, making incremental feature work increasingly complex. The team reframed these requests as a generic workflows platform in which Python functions served as actions, boolean logic as conditions, and workflows as graphs persisted in Postgres. After rejecting a dependency-based design, they built a graph-traversal engine, a Postgres-backed queue that retries asynchronous tasks, and an SDK for expressing constructs such as elif and else. The infrastructure now supports customer-facing policies and internal tasks; more than 45 million workflows have run, at over a million per day, while the core engine has not changed since launch. The project took weeks upfront but saved the engineering organization months by turning complex features into applications of broader abstractions.
Multiple Logs for Resiliency
An unexpected network outage can make an application misbehave while also removing crucial log data from the normal Fly.io pipeline. The post traces that pipeline from application STDOUT through Fly.io, NATS, the Log Shipper, Vector, and a third-party provider, then presents two ways to add redundancy. One strategy broadcasts framework logs to a separate file on each machine, preferably on a persistent volume with rotation; a Rails example uses ActiveSupport::Logger.broadcast. The other runs a separate app that subscribes directly to NATS, parses messages, and writes selected fields to disk, with a Fly authentication token and provisions for reconnection and error handling. File copies are more failsafe but raw and harder to fetch, while a custom shipper can produce more usable data but is less bulletproof.
More Predictable Shared Dyno Performance
Heroku describes a year-long effort to make CPU access more predictable for shared dynos in the Common Runtime. Previously, individual dynos could burst CPU use relatively freely when capacity was available, improving utilization but sometimes allowing misconfigured or malicious “noisy neighbors” to cause response-time spikes or H12 timeouts. The platform introduced improved resource isolation for eco, basic, standard-1X, and standard-2X Dynos, limiting bursts while preserving the ability to burst and aiming for fairer access to shared CPU resources. The change does not alter the number of dynos per instance, and Performance, Private, and Shield type Dynos are excluded because they run on dedicated instances.
Adventures in Garbage Collection: Improving GC Performance in our Massive Monolith
Shopify investigated high Ruby garbage-collection latency in its monolith, where minor marks took about 3 ms but major marks could exceed four seconds and some requests spent multiple seconds in GC. Logging showed that roughly 75% of major marks were triggered by oldmalloc and nearly 25% by shady, while nofree became dominant after oldmalloc was addressed. The team used production experiments, improved metrics, changed GC environment settings, increased initial heap slots, and added out-of-band collection during request processing. A 128 GB oldmalloc threshold reduced major marking by about 20%; later out-of-band collection cut the P99.99 and P99.9 GC-time tail by almost tenfold and largely eliminated median-request pauses. The resulting configuration, with collection frequency adapting from every 128 to every 512 requests, was deployed across the fleet after about three weeks.
Evolution of quality at Grab
As Grab expanded its superapp, manual QA sign-offs led to prolonged testing cycles and a peak in major production defects around 2019. To counter defect leakage and improve delivery pace, the engineering organization transitioned to a shift-left testing strategy across its software development lifecycle. Quality engineers began writing Given/When/Then acceptance tests prior to coding so that software engineers could execute them during development, accompanied by Definition of Ready and Definition of Done standards. The team also structured a multi-layer test strategy spanning unit tests, pipeline-executed UI component tests with mocked APIs, backend integration tests, and a custom internal end-to-end framework. Between 2019 and 2022, these changes reduced major and critical production issues by 60% and development-phase critical bugs by 40%.
The Case Against Monkey Patching, From a Rails Core Team Member
The post argues that Ruby monkey patches—global changes to existing objects or libraries—should be used sparingly because they are brittle, dangerous, and often unnecessary. They can make Rails and Ruby upgrades harder by depending on private APIs, leave applications exposed to vulnerabilities if security fixes are not mirrored, and accumulate poorly documented technical debt. One Shopify example, activerecord-pedant-adapter, changes Active Record’s MySQL2Adapter so database connections report MySQL query warnings; upstreaming that behavior into Rails allowed the gem to be archived and the patch removed. The recommended alternative is to investigate whether an upgrade or documentation resolves the issue, fix genuine defects upstream, and, when patching is unavoidable, document and test it while maintaining a removal plan.
Type Constraints in 65 lines of SQL
PostgreSQL’s extensible type system can encode validation and presentation behavior for a custom Semantic Versioning (SemVer) type. The article stores major, minor, patch, pre-release, and build-metadata components in a composite type, then wraps that type in a domain whose check constraint rejects null or negative numeric components and invalid array elements. A SQL helper validates nullable arrays with regular expressions, and the domain automatically enforces these rules when values are inserted into a table. For ergonomics, a function formats the composite value as a SemVer string, while a stored generated column persists that representation; the article also notes possible parsers, operators, and aggregation functions. It concludes that the ceremony is worthwhile when data integrity is critical and the specification is stable or changes infrequently.
The Hardest Part of Writing Tests is Getting Started
The article presents getting started with automated testing as the main obstacle, especially when a full suite requires helper tools, stubs, and fixtures that can overwhelm beginners. It argues that tests serve as executable documentation, preserve intended behavior and edge cases, and expose regressions before changes reach production. The recommended progression is to install a framework and verify both passing and failing tests, cover isolated functions, add stubs with sanitized API-response fixtures, and then build end-to-end tests while monitoring coverage. In an ETL example, API stubbing avoided expensive, slow calls, while database checks validated loaded and transformed data; the resulting tests covered more than 80 percent of the codebase and saved time for the author and team. The article also proposes E2E tests as a way to safely approach legacy code before refactoring it.
Performance Testing At Scale—for BFCM and Beyond
Shopify treats Black Friday Cyber Monday (BFCM) as its most critical period, and uses full-scale BFCM Scale Testing to assess whether the platform can handle peak traffic. Preparation combines capacity planning, resiliency testing, single-application performance testing, and repeated exercises that scale core services to their BFCM profiles before generating load. The hybrid architecture centers on Shopify Core, a monolithic Ruby application supported by services for storefront rendering, analytics, fraud analysis, payments, webhooks, and more. Five load flows cover browsing and buying, admin activity, flash sales, Storefront APIs, and Hydrogen and Oxygen, with Cronograma layering requests gradually and providing an abort switch. The exercises surfaced connection-count pressures, a Kubernetes ingress-nginx endpoint limit, and uneven load-balancer distribution; mitigations included refactoring, scaling dependent services, increasing pod capacity, and dedicated nginx node pools.
Catching Anomalies Early in Mobile App Releases
Nextdoor deploys weekly mobile updates across iOS and Android using phased rollouts starting at 1% adoption to minimize blast radius. Standard aggregate observability cannot reliably detect early regressions because early adopters skew significantly more active than average users, obscuring silent drops in overall metric noise. To overcome this selection bias, Nextdoor built App Release Anomaly Detection using difference-in-differences causal inference. The methodology verifies pre-adoption trends with standard deviation bounds and fits a linear regression model to estimate the effect against an unobserved counterfactual. During a rollout of iOS version v1.234.5, this model identified a statistically significant app session decline at 1% adoption, isolating the regression 10 days earlier than week-over-week metrics.
Automatically Rotating GitHub Tokens (So You Don’t Have To)
Shopify needed automated GitHub workflows that could clone from private repositories, update organization-level projects, and trigger downstream workflows but could not use GITHUB_TOKEN. Personal access tokens supplied those capabilities, yet introduced developer churn, user-attributed actions, and no organization-enforced rotation or expiry. The proposed system uses GitHub Actions and a GitHub App to generate repository secrets that rotate automatically while preserving PAT-like references in existing workflows. Its serverless design delegates scheduling, managed infrastructure, approvals, secret management, auditing, and break-glass operation to GitHub, while a custom TypeScript action encrypts and writes generated secrets. Prototyping exposed that parallel one-second jobs could accrue multiple billable minutes, prompting sequential execution in a consolidated workflow; the stated goal is improved resilience to developer churn and reduced leaked-secret blast radius.
What's new in Postgres 15?
PostgreSQL 15, released as a stable version in October 2022, adds security, performance, SQL, replication, logging, compression, and string-processing changes. CREATE permission is revoked from all users except the database owner, views can use caller permissions with security_invoker=on, and sorting is reported up to four times faster for in-memory and on-disk workloads; window functions also improve. The release introduces MERGE, supports two-phase commits and selective data in logical replication, allows conflicting transactions to be skipped or subscriptions disabled, and enables parallel workers for SELECT DISTINCT. It also adds JSON logging, Zstandard and ZL4 compression for WAL and pg_basebackup, and POSIX regexp functions such as regexp_substr() and regexp_count(). The post recommends upgrading for bug fixes and stability, while noting that migrated databases may need manual permission changes and upgrades should be checked for rare manual actions.
Point in Time Recovery is now available for Pro projects
Point-in-Time Recovery (PITR) is now available to eligible Pro Plan projects, expanding access beyond Enterprise customers and adding a Dashboard UI for enabling it. Projects created from August 2022 onward can enable PITR except in Northeast Asia / Seoul, and they must use the Small compute add-on; older projects are not yet generally eligible. The system combines daily physical database snapshots with archived Write Ahead Log (WAL) records, using the open-source WAL-G tool; snapshots are uploaded daily and WAL files are archived as transactions occur. WAL files are backed up at two-minute intervals, potentially faster when a file threshold is reached, reducing the recovery point from the up-to-24-hour exposure of daily backups. PITR backups retain up to seven days by default, with self-serve expansion to 28 days, and restoration can target a specified point in time.
Planning in Bets: Risk Mitigation at Scale
Preparing for BFCM and other high-traffic events, Shopify describes risk mitigation for a globally distributed, complex, interconnected infrastructure platform that peaked at 75.98M requests per minute, or 1.27M per second. At this scale, identifying and mitigating every possible failure is impossible, so the process starts with “what could go wrong” exercises, then uses voting, expert review of likelihood and severity, and assigned ownership to prioritize action. Decision makers are empowered to weigh risks and rewards, while summarized findings, including key risks and single points of failure, are shared to maintain alignment and awareness. The post frames engineering choices as probabilistic bets: decision quality should be judged separately from outcomes, and uncertainty cannot be eliminated, only addressed through thoughtful prioritization and communication.
Caching Without Marshal Part 1: Marshal from the Inside Out
Rails caching commonly relies on Ruby’s Marshal serialization, which can encode complex objects but also embeds their class names in cache entries. Shopify describes an incident in which a beta-flag refactor changed classes while old and new code overlapped during deployment; old code then failed on cached instances containing unfamiliar class names and methods, despite passing CI. The article examines Marshal through Ruby’s marshal.c implementation, covering atomic and composite types, instance variables, object references, circularity via TYPE_LINK, and core-type subclasses via TYPE_UCLASS. It presents MessagePack as a more compact, stricter, and controllable alternative intended to make caching safer by default, while deferring the cache migration and implementation details to the series’ next part.
The Management Poles of Developer Infrastructure Teams
Developer infrastructure managers must balance three poles: management support, system and domain expertise, and organization-wide road maps, with the tension intensified by teams that often lack dedicated product managers and support critical systems for hundreds or thousands of users with six to eight developers. Domain-focused teams build deep knowledge needed for maintenance, migrations, incident investigation, on-call coverage, and developer-experience improvements, but their local road maps can diverge from the department’s highest-impact opportunities. The proposed responses are temporary individual assignments, having a whole team contribute to another team’s goals, and forming a cross-functional tiger team; each shifts trade-offs among focus, maintenance, expertise, motivation, and management oversight. A month-long tiger team built the first Spin proof of concept, which led to a dedicated team, but no structure eliminates the tensions, so choices must be judged according to context.
Hubble: Our Tool for Encapsulating and Extending Security Tools
Shopify’s Trust team faced rising complexity in securing a distributed device fleet, including many tools, provisioning and patching work, varied networks, and the loss of in-person remediation. Hubble was built as a unified layer over mobile device management and other security systems, ingesting and standardizing their data while sending commands back with granular access controls and centralized auditing. IT staff use it for inventory, device management, and security, while employees can view device health and compliance, receive remediation guidance, and manage test-device or beta participation. The platform also serves as a standardized source for automation, and the post presents encapsulation and cross-team investment as ways to reduce administrative overhead, support proactive security, and limit risk.
Safely Adding NOT NULL Columns to Your Database Tables
Shopify’s Database Migrations team investigated whether Large Hadron Migrator (LHM) can safely add NOT NULL columns to existing MySQL tables with minimal downtime. LHM creates a shadow table, installs triggers for INSERT, UPDATE, and DELETE operations, and copies records in batches before renaming the tables. The experiments varied whether the new column had a DEFAULT value or UNIQUE index and whether MySQL used strict or non-strict mode, evaluating trigger compatibility and record counts. A NOT NULL column with a DEFAULT was backward compatible, but adding a UNIQUE index could cause data loss; without a DEFAULT, strict mode broke compatibility while non-strict mode used implicit defaults, which may be undesirable.
Six Tips for Staying Technical as a CTO
The article examines how a CTO can remain technically credible after moving from the individual contributor track into leadership. It argues that leadership inevitably reduces technical depth because performance reviews, one-on-ones, organizational communication, and other people responsibilities limit time in code, making the transition an intentional choice for those who value managing people. Its recommended practices include voracious reading, coding as a hobby, participating in hackathons, learning directly from more technical colleagues, maintaining a private network of technical advisors, and aggressively clearing bugs, incident-remediation work, and backlog. The author illustrates hands-on learning with Shopify’s Spin environment, which provisions a cloud development instance and opens a repository in about 30 seconds, and concludes that controlling operational issues preserves time for innovation and technical leadership.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.