Copy-and-Replay vs Branch-at-a-Point: Two Backup Recovery Paths
A side-by-side comparison of copy-and-replay recovery and branch-at-a-point restores, showing why restore time scales with data size in one model and not the other.
Loading…
Topic
Reliability
286 posts about Reliability. Every summary links to the original.
- 286 posts
- 45 companies
- 2 explainers
See the idea
Reliability explainers
The SQS Decision Most Diagrams Skip: Normal Work vs. Broken Work
Amazon SQS connects producers and workers, but its real design test is failure. Compare the processing path with the signals and controls needed when queue operations break.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Towards Model-based Verification of a Key-Value Storage Engine
MongoDB developers formalized the interface boundary between their distributed transactions protocol and the underlying WiredTiger key-value storage engine using a compositional TLA+ specification. This modular model established an abstract contract to verify that the storage engine's concrete timestamp-based operations matched the semantics expected by the distributed transaction layer. To check conformance, a modified version of the TLC model checker generates a complete graph of reachable states for finite parameters and computes path coverings. Each identified path is translated into an automated test case consisting of a sequence of storage engine API calls. For a finite model with two keys and two transactions, this workflow generated 87,143 test cases that executed against WiredTiger in approximately 40 minutes.
Building a mental model for Stripe payments
An online bookshop checkout must coordinate card networks, issuing banks, fraud checks, regulatory requirements, currencies, and varied payment methods while completing in roughly 2–3 seconds. Stripe abstracts these concerns through a unified API, with the PaymentIntent representing the payment lifecycle from amount and currency collection through tokenization, authorization, capture, and settlement. The Payment Element and client-side tokenization keep sensitive card details away from the shop’s servers, while Stripe may perform risk checks and additional 3D Secure authentication. Because payments can change state minutes, hours, or days later through refunds and disputes, webhook events should be treated as the source of truth for synchronizing order and downstream systems without polling.
IBM and UC Berkeley Diagnose Why Enterprise Agents Fail Using IT-Bench and MAST
IBM Research and UC Berkeley analyze how agentic LLM systems fail in enterprise IT automation, including incident triage, logs and metrics queries, and Kubernetes actions across long-horizon tool loops. To move beyond benchmark success rates, the analysis applies MAST (Multi-Agent System Failure Taxonomy) to execution traces, converting failures into structured signatures across system design, inter-agent misalignment, and task verification. It covers 310 ITBench SRE traces from Gemini-3-Flash, Kimi-K2, and GPT-OSS-120B, whose mean recalls were 75.5%, 28.6%, and 12.4%, respectively. Gemini averaged 2.6 failure modes per failed trace, compared with 4.7 for Kimi and 5.3 for GPT-OSS-120B, while FM-3.3, Incorrect Verification, was identified as the strongest predictor across models; the analysis recommends external verification, explicit termination controls, and clarification branches.
Supabase incident on February 12, 2026
Supabase reports a 3-hour-and-42-minute outage on February 12–13, 2026, affecting every service in its us-east-2 region, including Postgres, Auth, Data APIs, Edge Functions, Storage, and Realtime. A deployment of an internal monitoring service enabled AWS VPC Block Public Access in block-bidirectional mode, blocking internet gateway traffic across production VPCs in the region. The incident response was delayed because alerts pointed toward shared services, the relevant CloudTrail event was inconspicuous, pre-production lacked us-east-2, and infrastructure specialists were not involved initially. Supabase restored connectivity by rolling back and removing the monitoring stack, then audited regions, added AWS Organizations policies and IaC safeguards, and planned account isolation, external connectivity probes, production-parity testing, and improved escalation.
OpenEnv in Practice: Evaluating Tool-Using Agents in Real-World Environments
OpenEnv is an open-source framework from Meta and Hugging Face for evaluating AI agents against real systems rather than simulations, with stateful environments that connect to tools and APIs. Turing's production-grade Calendar Gym applies this model to calendar operations, combining access control lists, partial visibility, temporal reasoning, and multi-step workflows in isolated sessions. Agents often handled individual actions but became less reliable on longer or ambiguous tasks: success was close to 90% with explicit calendar identifiers and roughly 40% when requests used natural-language descriptions. More than half of failures involved malformed arguments or incorrect action ordering despite selecting the correct tool, while structured validation, permission, and datetime errors provided concrete recovery signals.
Engineering at scale, with purpose.
Senior software engineer Brett Shouse outlines site reliability engineering initiatives and infrastructure modernization efforts underway at Thumbtack. The engineering organization is currently leading an operational project to migrate multiple self-hosted observability services onto a single, unified SaaS platform. This architectural transition consolidates application logs, distributed traces, and system metrics into one accessible view tailored for engineers, customer support staff, and company executives. Moving away from legacy self-hosted monitoring systems reduces systems administration overhead, mitigates alert fatigue, and lowers direct operational infrastructure costs. Concurrently, the reliability team tackles technical debt accumulated from rapid organizational growth by establishing structured incident response processes and automating repetitive operational toil.
Edge AI Made Easy: MongoDB and ObjectBox Data Synchronization
Modern edge AI deployments require immediate real-time decision-making, local privacy, and offline operation while retaining the analytical scale of central cloud infrastructure. To address these hybrid demands, MongoDB and ObjectBox partnered to release the ObjectBox MongoDB Sync Connector as part of ObjectBox 5.0. ObjectBox operates as a lightweight on-device database with local vector search, multi-language bindings, and high-throughput I/O suited for resource-constrained hardware. In environments such as industrial factory floors and restaurant point-of-sale terminals, devices capture and evaluate operational data onsite without depending on active network connections. The native connector subsequently synchronizes critical operational datasets to MongoDB Atlas for long-term storage, fleet-wide monitoring, and machine learning model retraining.
The safety of speed: How we ship code 180 times per day
Intercom describes how it averages 180 production ships per workday in January 2026 while targeting 99.8+% availability, with an average 12-minute merge-to-production cycle. The post focuses on its Ruby on Rails monolith, deployed to three data-hosting regions through independent pipelines, and frames small batches as a way to limit accumulated risk. An automated pipeline builds a deployable slug, runs parallel CI, verifies it in pre-production with boot, CI, and Datadog Synthetics gates, then performs a staggered rollout with process-level rolling restarts. Safety also depends on engineer ownership, feature flags, production testing on a single host, outcome-focused heartbeat metrics, and rapid automatic or manual rollbacks. Incident reviews then harden the system, supporting the conclusion that speed is used to keep changes small, feedback immediate, and recovery fast rather than to trade stability for velocity.
Configuring Stripe using Terraform and AI agents
The post addresses the difficulty of using AI agents to configure Stripe resources when one-off API calls leave no durable source of truth for state, changes, or environment differences. It recommends using agents to author Terraform configuration rather than operating Stripe directly, so products, prices, and webhook endpoints become explicit, reviewable code. The example defines a Standard Plan with $20 monthly and $200 annual prices, selected webhook events, and outputs for both price IDs, then applies it with terraform plan and terraform apply. Terraform provides declarative, repeatable configuration supported by Git history, pull-request review, and environment-specific workspaces. Separate sandbox and livemode workspaces, paired with matching STRIPE_API_KEY values, are presented as a way to reduce accidental cross-environment changes.
AssetOpsBench: Bridging the Gap Between AI Agent Benchmarks and Industrial Reality
AssetOpsBench is an evaluation framework designed to benchmark agentic AI systems within industrial Asset Lifecycle Management operations, such as chillers and air handling units. The benchmark incorporates 2.3 million sensor telemetry points, 4,200 work orders, 53 structured failure modes, and over 140 curated scenarios spanning four agent roles. Systems are assessed across six qualitative criteria, including task completion, retrieval accuracy, result verification, sequence correctness, clarity and justification, and hallucination rate. Evaluation traces are analyzed using a trajectory-level diagnostic pipeline called TrajFM, which extracts and clusters failure patterns. In community tests covering over 300 agents, no evaluated model family reached the 85-point readiness threshold.
AprielGuard: A Guardrail for Safety and Adversarial Robustness in Modern LLM Systems
Modern language model systems face security and safety threats spanning prompt injection, jailbreaks, and tool manipulation across long-context agentic workflows. To address these vulnerabilities, researchers introduced AprielGuard, an 8B parameter causal decoder-only transformer based on a downscaled Apriel-1.5 Thinker Base model. The model detects 16 safety risk categories alongside binary adversarial attacks across standalone prompts, multi-turn dialogues, and agentic workflows. Trained on synthetic datasets created with Mixtral-8x7B, NVIDIA NeMo Curator, and SyGra, AprielGuard operates in both explainable reasoning and fast non-reasoning modes. Evaluation shows high precision and recall on safety and adversarial benchmarks, though trade-offs exist between explainability latency and non-reasoning performance.
From Python3.8 to Python3.10: Our Journey Through a Memory Leak
During an initiative to upgrade Python services from version 3.8 to 3.10, Lyft engineers encountered severe latency spikes and timeouts in one service within their test environment. Profiling DynamoDB queries revealed that gevent thread joins were taking up to 30 seconds while pod memory consumption climbed steadily. To isolate the issue, engineers used an internal tracemalloc-based profiler triggered via SIGUSR2 signals, temporarily disabling Gunicorn preload to prevent workers from terminating prematurely on signal receipt. Memory traces pointed to botocore and an incompatibility between weakref.finalize and gevent monkey patching in urllib3 version 1.26.16, which prevented connections from returning to the pool. Downgrading urllib3 to version 1.26.15 immediately resolved both the timeouts and the memory leak before a permanent fix arrived in gevent and urllib3 updates.
Versioning in Stripe Workflows: Ship workflow changes with confidence
Stripe Workflows previously required teams to edit live workflow definitions directly, leaving no draft state, historical definitions, straightforward rollback, or reliable way to inspect which configuration shaped an older run. Versioning addresses this with a dedicated draft for iteration, immutable numbered published versions, and version history that can include descriptions, authors, status, and run counts. Publishing creates a complete snapshot containing steps, branches, conditions, variable references, calls, and configuration; new runs use it immediately, while in-progress runs finish on the version they started with. Each run records its exact version and exposes the workflow definition, including taken and untaken branches, while a past version can seed a new draft for rollback; versioning is available in Stripe Workflows, where new workflows include a draft by default.
Building production-ready Stripe subscriptions using Kiro powers
Kiro's Stripe power is presented as a way to build a production-ready SaaS subscription flow while avoiding the fragmented output common with generic coding assistants. Using a natural-language specification for three monthly tiers and a three-month one-time option, Kiro creates a React frontend, Node.js backend API, CDN-hostable static assets, and server-side Stripe Checkout Session creation. The integration includes Stripe-specific steering files, MCP servers, and hooks, while the webhook handler preserves the raw request body, verifies HMAC signatures with stripe.webhooks.constructEvent(), rejects invalid events with HTTP 400, and routes lifecycle events such as checkout completion and subscription cancellation. The post concludes that this provides a secure, scalable foundation while leaving teams to customize business logic, including access provisioning and deprovisioning, instead of building payment infrastructure from scratch.
Engineering the right opportunities for Thumbtack Pros.
Ashmann Syngle, a backend software engineer on Thumbtack's Pricing team, focuses on systems that connect local service professionals with homeowners. To improve matching and revenue operations, the team recently deployed new pricing features, launched experiments, and partnered closely with Data Science and Monetization Experience groups. Current engineering efforts center on infrastructure enhancements, specifically upgrading alerting and monitoring across owned services to raise system reliability and operational efficiency. Because the team maintains complex monetization systems within a codebase that has evolved over many years, engineers conduct deep system analyses to evaluate edge cases before implementing foundational platform modifications. Thumbtack supports these initiatives through a virtual-first operational model complemented by regular in-person offsites.
Run for your money: Engineering a treadmill that prints store credit
For the 2025 NYC marathon weekend, Shopify and Endorphins Running built a gamified treadmill that awarded in-store credit for maintaining a self-selected pace. Participants had 30 seconds to reach their target, then earned credit for each second spent within a defined pace zone, with a personalized barcode receipt redeemable through a Shopify POS UI extension. The implementation used a curved, motorless TrueForm Trainer, FTMS messages, a Python relay, Socket.IO, a local Remix application, receipt printing, and a production-hosted POS integration. Interval timestamps were normalized because treadmill messages arrived irregularly, while a state machine coordinated warm-up, gameplay, operator controls, database updates, discounts, and printing. The resulting experience attracted new customers, with 63% of participants identified as new customers, and its patterns could support other sensor-driven in-store incentives.
How we prepare Shopify for BFCM
Shopify’s BFCM readiness program addresses the challenge of operating critical commerce infrastructure at traffic levels informed by 2024 records, while preparing APIs with no prior holiday-traffic history. Starting in March, teams combined capacity planning on Google Cloud, an infrastructure roadmap, risk assessments, chaos-engineering Game Days, and a Resiliency Matrix covering failure modes, recovery objectives, runbooks, and on-call coverage. Genghis load tests and Toxiproxy network faults ran across three GCP regions, while five April–October scale tests exercised browsing, checkout, analytics, webhooks, regional failover, and cascading failures; testing reached 146 million requests per minute and a p99 target of 200 million. The work exposed and addressed Kafka partition, memory, timeout, routing, queueing, and failover issues, while the operational plan adds monitoring, 24/7 incident coverage, merchant communications, live tuning, and a post-BFCM review.
We’re Hiring a Principal Backend Engineer to Shape the Future of Medium
Medium is hiring a Principal Backend Engineer, its most senior individual-contributor level, to lead the evolution of backend systems supporting reading, writing, and discovery for millions of users. The role combines hands-on engineering with organizational leadership: the hire will define backend strategy, set quality and architectural standards, review critical designs and code, mentor engineers, and partner with engineering, design, data, and executive leadership. Its technical focus includes modernizing a large living codebase, improving reliability and scalability, and balancing long-term platform evolution with building, debugging, and refining systems directly. Medium says the position should help teams move faster and ship with confidence while keeping the technology fast, stable, and built for longevity; it is fully remote in the US and seeks engineers experienced with systems used by millions, technical debt, fan-outs, and complex edge cases.
The 10 Skills I Was Missing as a MongoDB User
Transitioning from relational databases to MongoDB often leads developers to replicate SQL habits, causing severe performance bottlenecks, brittle schemas, and operational challenges. Effective document design requires balancing referencing and embedding based on query access patterns and update frequencies, rather than placing every entity into separate collections or excessively nesting large documents. Database performance depends on matching indexes to specific query shapes and field orders, which developers can systematically verify by inspecting execution plans using explain plans. Utilizing the built-in aggregation framework allows filtering, grouping, and calculations to run natively inside the database, eliminating slow, manual data transformations in application code. Establishing early monitoring for latency, replication lag, memory usage, and cluster failover resilience ensures stability before adopting advanced tools like Atlas Search and Vector Search.
Carrying Complexity, Delivering Agility
MongoDB centers its engineering architecture around resilience, intelligence, and simplicity to minimize developer cognitive and operational burdens when building distributed applications. Security is enforced through dedicated clusters in isolated virtual private networks alongside Queryable Encryption, which allows equality and range queries on ciphertext without decryption keys ever leaving the client. High availability is built on replica sets across independent availability zones and multi-cloud topologies, utilizing consensus mechanisms that commit writes only after majority acknowledgment in the active term. Operational friction in artificial intelligence workloads is addressed by integrating vector search directly into the core query engine, eliminating brittle extract-transform-load pipelines and separate vector databases.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.