Copy-and-Replay vs Branch-at-a-Point: Two Backup Recovery Paths
A side-by-side comparison of copy-and-replay recovery and branch-at-a-point restores, showing why restore time scales with data size in one model and not the other.
Loading…
Topic
Reliability
286 posts about Reliability. Every summary links to the original.
- 286 posts
- 45 companies
- 2 explainers
See the idea
Reliability explainers
The SQS Decision Most Diagrams Skip: Normal Work vs. Broken Work
Amazon SQS connects producers and workers, but its real design test is failure. Compare the processing path with the signals and controls needed when queue operations break.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
vLLM V0 to V1: Correctness Before Corrections in RL
PipelineRL migrated its rollout inference engine from vLLM V0 (0.8.5) to V1 (0.18.1), but trainer-side logprobs, policy ratios, KL, entropy, clip rate, and reward diverged from the V0 reference. The investigation separated semantic mismatch, inference-path mismatch, and objective mismatch, then prioritized backend parity before changing the RL objective. Fixes included processed rollout logprobs, explicit V1 runtime settings disabling prefix caching and async scheduling, a V0-like inflight weight-update path, and an fp32 lm_head for the final projection. With those changes, the final V1 run tracked the V0 trajectory across the reported metrics, while processed_logprobs alone did not resolve the training gap.
amazon.science
Navigating uncertainty in Amazon's middle-mile network
Amazon's middle-mile logistics network coordinates tens of millions of packages across hundreds of facilities while managing subtle demand variations and major operational disruptions. To solve this mixed-integer optimization challenge without combinatorial explosion, Amazon models routes at coarse time resolutions augmented by precomputed 15-minute timing bounds. The system uses Monte Carlo methods alongside a graph attention network featuring interconnected site graphs and origin-destination graphs to capture complex spatial correlations and flow interdependencies. By generating synthetic flow permutations, planners stress-test candidate network designs against hundreds of plausible scenarios to prioritize routing optionality. This risk-aware modeling approach preserves network flexibility, absorbs severe shocks, and yields potential cost savings while safeguarding delivery promises.
Introducing Application Metrics: Track the signal, see the spike, jump to the trace
Sentry launched Application Metrics to capture application signals without pre-aggregating counters or discarding high-cardinality attributes like user IDs, project IDs, and regions. The system supports counters, distributions, and gauges within the existing SDK while linking each metric event directly to trace IDs for integrated debugging. Engineers demonstrated the feature by troubleshooting a Session Replay failure that triggered silently when sessions exceeded 1,000 video segments. By instrumenting distributions for replay event counts with project and replay attributes, the team isolated seven affected occurrences, navigated to connected traces and log lines, and reproduced the issue. Application Metrics comes enabled in recent SDK versions with 5GB included per plan.
How GitHub uses eBPF to improve deployment safety
Deployment scripts can introduce dangerous circular dependencies when they rely on services or assets from platforms that are currently experiencing outages. Blocking network access at the host level is impractical because stateful nodes continue serving live traffic during rolling deployments. To solve this, GitHub isolates deploy scripts into dedicated Linux cGroups and attaches custom eBPF programs via the cilium/ebpf Go library. The system uses socket-address hooks to redirect DNS queries to a userspace proxy that checks a domain blocklist, while egress packet hooks map DNS transaction IDs to process IDs. This approach successfully prevents deploy-time circular dependencies, provides full command-line audit logs for blocked requests, and speeds up incident recovery.
How we built Stripe Credits: A programmable, auditable way to pay your Stripe fees
Stripe Fee Credits replaces ad-hoc discounts and manual cash adjustments with a programmable, auditable virtual balance dedicated to eligible Stripe fees. The architecture has three layers: a standalone gRPC credits engine for metadata, policies, lifecycle, and audit history; logical financial accounts built on Money Movement and Storage; and fee-settlement orchestration that applies eligible credits before the primary balance. Logical accounts carry ledger balances without representing real cash, while platform checks restrict Originated Money Transfers to logical-to-logical movements and exclude these accounts from cash sweeps. Prepaid and Stripe-funded credits share the user experience but use separate funds flows: prepayments create deferred revenue and collect tax at drawdown, whereas Stripe-funded credits are booked as Stripe costs with taxes written off. The resulting infrastructure supports incentives, prepaid annual fees, and other programs across 26+ countries, with Dashboard balances and real-time auditability, while avoiding bespoke engineering for each commercial model.
We're Transferring the Stripe Sync Engine to Stripe
The Stripe Sync Engine repository is moving from supabase/stripe-sync-engine to stripe/sync-engine after growing from an open-source tool for syncing Stripe data into a Stripe schema in Postgres. The transfer follows collaboration that added cursor-based incremental sync, JSONB storage with generated columns, multi-account support for Stripe Connect platforms, event ordering in a new CLI, and automatic incremental backfilling using Supabase Cron and Queue. The latest release adds coupon synchronization, branching support, immediate syncing on installation, one-click upgrades, SSL enforcement, and tighter installation controls, along with clearer errors and dashboard visibility for Edge Function source code. The project remains open source under Apache 2.0, existing installations continue to work, and the Supabase dashboard integration is unchanged. With Stripe data in Postgres, users can join it with application data for subscription checks, feature gating, MRR, and churn analysis.
Managing context in long-run agentic applications
Long-running multi-agent systems struggle with context management because accumulating raw message histories degrades inference quality, increases latency, and exceeds context window limits. In a collaborative security investigation platform, passing unrestricted history can also introduce confirmation bias across specialized agents. To maintain coherence across unbounded rounds of investigation, the system eliminates raw message history carryover between invocations. Instead, it coordinates agents through three structured context channels: a Director's Journal for orchestration memory, a Critic's Review that scores findings to filter hallucinations, and a Critic's Timeline of validated chronological events. This architecture provides agents with tailored context without overwhelming their inference capacity.
Re-imagining ML Serving Infra: My Winter Internship at Ramp
Ramp’s ML Platform team needed to separate flexible model development in ml-pipelines from the Intelligence Platform’s 99.9%+ uptime and low-latency requirements. The internship project, Bring-Your-Own-Model-Weight Serving (mw-serve), built a dedicated serving layer that pulls model weights from MLflow and a training-environment base image from ECR, then runs each model in an isolated instance. To protect uptime, it added deployment safeguards, pre-traffic liveness checks, and monitors, while latency work instrumented client TCP/TLS connections and server processing. The investigation traced extreme tail latency to connection churn, which improved with longer keepalives and startup warmups, and to garbage-collection pauses associated with memory drops; tuning thresholds and freezing objects removed more outliers. By the final day, one Intelligence Platform model was serving production traffic through mw-serve, with additional models in shadow testing.
Selective Test Execution at Stripe: Fast CI for a 50M-line Ruby monorepo
Stripe’s Ruby monorepo contains about 50 million lines, 100,000 test files, and roughly 1.2 million test units, making sequential execution impractical at an estimated four months. To keep CI within minutes, Selective Test Execution (STE) runs tests whose recorded file accesses intersect files changed since a suitable baseline, averaging about 5% of the suite and under 10% of always-run compute. A dynamically linked C++ library loaded with LD_PRELOAD intercepts file opens, associates them with hierarchical test scopes, and propagates tracing to child processes; logs are aggregated into a selection index outside the syscall hot path. Guardrails cover tests affected by directory discovery, previously failing tests, selective linting, generated files, and reproducible database-backed baselines ordered by Monotonic Revision IDs. The system is presented as a way to preserve safety and affordability while scaling CI for a highly dynamic language.
How Eisan made POS analytics faster, cheaper, and more reliable with ClickHouse Cloud
Eisan System Development moved a retail analytics workload from a QlikView-based platform to ClickHouse Cloud after rising ID-POS data volumes caused slower processing, failed analyses, and increasing licensing and server costs. Client records from supermarkets and drugstores support basket analysis, cross analysis, and real-time personalization, with some datasets nearing 200 million records annually. The team compared open-source options including PostgreSQL and YugabyteDB, then tested ClickHouse configurations, including a 30-core, 120-GB setup, while accounting for cache memory usage through load testing. In comparison, ClickHouse ran on slightly lower-spec hardware under high-load conditions without observed performance deterioration, and Eisan reports no cases so far where it failed to return a result; the team is now targeting 100 billion records and exploring off-hours processing and scheduled scaling.
Zero downtime Upgrade: Yelp’s Cassandra 4.x Upgrade Story
Yelp's Database Reliability Engineering team upgraded over a thousand Cassandra nodes from version 3.11 to 4.1 without downtime. The upgrade delivered reliability, observability, and performance improvements, including Java 11 support, configurable guardrails, and fixed incremental repairs. To avoid the high costs, consistency compromises, and data streaming delays of standing up separate datacenters, the team selected an in-place rolling upgrade strategy. The rollout orchestrated staged node updates alongside compatible versions of the Stargate proxy and custom data pipeline connectors without requiring client code modifications. Although Stargate 2.x introduced regressions on range and multi-partition queries that required downgrading to 1.x, the finished migration achieved up to a 58% reduction in p99 latency across key clusters.
Frontend Engineering at Palantir: Building a Backend-less Cross-Application API
Palantir’s Application Frameworks team builds the “backend of the frontend”: frameworks that let applications communicate during workflows spanning multiple interfaces. The work addresses environments with denied, disrupted, intermittent, or limited (DDIL) connectivity, where relying on backend services can introduce unreliable latency. The team generalized the Gaia Controller prototype into Shared State, which lets applications define named data models and exchange updates organized by Media Type, and Commands, which exposes arbitrary logic through command IDs and request objects that return responses. Both frameworks use frontend-only browser APIs for client-side interaction, support workflows such as pairing Gaia with Workshop, and are also being used as tools by AIP agents; the post says adoption is growing across Palantir.
Improving storage efficiency in Magic Pocket, our immutable blob store
After introducing a direct erasure-coding service, Dropbox encountered severe storage fragmentation in its immutable blob store, Magic Pocket, due to a large number of under-filled volumes. Because their legacy compaction strategy was only designed to top off near-full volumes, the team created new multi-strategy compaction methods to aggressively consolidate sparse volumes. These approaches leverage dynamic programming and streaming pipelines to rapidly reclaim unused disk capacity at exabyte scale.
From Custom to Open: Scalable Network Probing and HTTP/3 Readiness with Prometheus
Slack encountered a lack of client-side observability when introducing HTTP/3 at its edge. Because HTTP/3 runs over QUIC on UDP instead of TCP, neither existing commercial SaaS tools nor the internal Prometheus Blackbox Exporter could probe the new endpoints. To address this gap, an engineering intern integrated the open-source quic-go library into Prometheus Blackbox Exporter to build a native, configurable HTTP/3 probe. Slack deployed an internal implementation using the upstream functionality while waiting for pull request reviews, ultimately landing the contribution upstream. The resulting setup provides a unified view of HTTP/1.1, HTTP/2, and HTTP/3 metrics within Grafana, enabling reliable alerting and faster correlation with other telemetry.
Boost Training Goodput: How Continuous Checkpointing Optimizes Reliability in Orbax and MaxText
Orbax and MaxText introduce continuous checkpointing to balance training reliability and performance when hardware failures or preemption events interrupt long-running jobs. Fixed checkpoint intervals create a trade-off: infrequent saves risk substantial lost work, while frequent saves can block or bottleneck training, particularly over unstable networks. MaxText enables asynchronous checkpointing that attempts a new save only after the previous background request is fulfilled; its continuous mode overrides checkpoint_period, and max_num_checkpoints_to_keep can limit storage use. A llama-3.1-70B benchmark on two v5p-128 slices found smaller P50 checkpoint intervals, but average training step time increased because of more frequent device-to-host transfers; the analysis estimated resource conservation against mean-time-between-failure assumptions. Orbax also supports minimum intervals and customizable preservation policies, while co-locating storage with training is recommended because cross-metro bandwidth can reduce checkpointing speed and reliability.
Navigating Regional Network Blocks
Supabase describes how regional network blocks disrupted access to its multi-tenant developer platform and shares three incidents from September 2025 through March 2026. It explains that DNS-level or transport-layer blocking can affect every tenant when projects share domains and IP addresses, while adding a hosting domain to the Public Suffix List provides only partial mitigation. In the UAE, a project distributing unauthorized streaming content triggered an approximately 18-day block that ended after removal and engagement with the TDRA. Yemen saw TCP connections dropped for about eight days with no official explanation, while India’s roughly eight-day restriction was rescinded after project enforcement and technical engagement with MeitY. The post says providers should build regulator contacts, strengthen abuse response pipelines, and develop alternatives such as proxy domains, custom domains, and other workarounds.
5 Career Myths From Women in Engineering
Three engineering leaders at Two Sigma—Angela Wang, Cecilia Ye, and Mae Santos—challenge conventional career progression models by examining five persistent industry myths. Diverse transitions across civil engineering, enterprise architecture, and customer-facing solutions architecture supplied foundational skills for modern roles in feature engineering, reliability engineering, and real-time trading optimizers. The leaders emphasize that deep domain expertise stems from direct proximity to an active system rather than cumulative years of tenure. Beyond core programming capabilities, proactive contributions such as upgrading operational standards and cultivating peer relationships drive career opportunities and leadership recognition. Adapting to emerging technologies and maintaining consistent execution allows engineers to build credibility across evolving technical environments.
Unordered mode for GCS ClickPipes is now available
ClickHouse Cloud has added unordered mode to its Google Cloud Storage (GCS) ClickPipes connector, enabling continuous data ingestion regardless of file naming or arrival sequence. Standard GCS ClickPipes rely on polling every 30 seconds and require files to arrive in strict lexicographical order, which causes out-of-order data and backfills to be ignored. Unordered mode replaces polling by consuming OBJECT_FINALIZE notifications from Google Cloud Pub/Sub whenever objects land in a bucket. State tracking is managed through a metadata store, and the connector automatically reprocesses batches upon failure while maintaining exactly-once insertion guarantees. Setup requires configuring a Pub/Sub topic and bucket notifications via tools like gcloud, followed by configuring the ClickPipe via the console or Terraform provider.
Supabase Storage: major performance, security, and reliability updates
Supabase Storage has shipped updates live in all regions for projects using Storage, targeting performance at scale, security vulnerabilities, and reliability issues. As projects reached tens of millions of objects, concurrent uploads exposed write bottlenecks, certain SQL deletes could orphan files, and a crafted path could escape the file backend’s storage root. The object-listing rewrite removes the prefixes table, six triggers, and twelve helper functions, derives folder structure from the objects table with a hybrid skip-scan algorithm, and uses cursor-based pagination; deep pagination on 60 million rows is up to 14.8 times faster without a write penalty. Direct SQL deletes are blocked by default through a statement-level trigger, while a configurable 30-second timeout supports cancellation of disconnected clients via native Postgres protocol. Idempotent migrations, TUS lock-race prevention, orphan-scanner fixes, OpenTelemetry metrics, and adapter and API bug fixes are included, with no code changes required.
PlanetScale protects production reliability with Bugbot
PlanetScale describes code review as the bottleneck created by rapidly increasing agent-generated code, while human review capacity remained fixed. To preserve production reliability, the company adopted Bugbot as a dedicated agentic review layer alongside its existing process. Bugbot surfaces semantic and logical issues that static analyzers, linters, and straightforward frontier-model reviews may miss, including premature completion states, non-converging asynchronous controllers, and database restart edge cases. Across more than 2,000 pull requests reviewed monthly, roughly 80% of Bugbot's comments are addressed before merge. PlanetScale says this prevents potential production downtime, saves the equivalent of two full-time engineers of review effort, and lets engineers focus on complex infrastructure work without sacrificing quality.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.