Copy-and-Replay vs Branch-at-a-Point: Two Backup Recovery Paths
A side-by-side comparison of copy-and-replay recovery and branch-at-a-point restores, showing why restore time scales with data size in one model and not the other.
Loading…
Topic
Reliability
288 posts about Reliability. Every summary links to the original.
- 288 posts
- 46 companies
- 2 explainers
See the idea
Reliability explainers
The SQS Decision Most Diagrams Skip: Normal Work vs. Broken Work
Amazon SQS connects producers and workers, but its real design test is failure. Compare the processing path with the signals and controls needed when queue operations break.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
POSETTE Talk Recap - Postgres Isn't Slow. Your Storage Is
At POSETTE 2026, Sai Srirampur’s talk examines how storage limits can produce slow ingestion, erratic P95 reads, lagging autovacuum, slower checkpoints, and logical replication lag in large PostgreSQL workloads. The benchmark ran the same 3.3-billion-row, random-UPDATE workload on four local-NVMe clusters and four baseline gp3 EBS clusters, using m6id.4xlarge instances, 64 clients, and 16 worker threads. NVMe reached median 16,030 TPS versus 1,734 TPS on EBS, reduced median latency from 36.9 ms to 4.0 ms, and completed the tested VACUUM and replication-backlog tasks faster. The analysis attributes most of the gap to storage waits rather than CPU work, while the proposed production model combines local NVMe with quorum-based replication, external base backups, and archived WAL for recovery.
How we knew COVID was over (and what our models had to unlearn)
Airbnb's forecasting team outlines their framework for managing production models that experience drift or external shocks. Rather than treating model updates as a generic retraining task, they distinguish between refitting with new data, respecifying the underlying model structure, and intentionally holding without changes. This discipline prevents teams from chasing temporary noise, carrying obsolete crisis assumptions, or over-engineering solutions during disruptions.
Improving infrastructure efficiency for growing demand in the age of AI
Dropbox manages growing infrastructure demands from AI and standard workloads by maximizing the efficiency of its existing data centers rather than just building more capacity. Through a system-level approach spanning software, hardware, and physical facility design, the company dynamically manages power and balances workloads. These optimizations, combined with higher storage density and data-driven hardware lifecycle management, have significantly reduced energy use per unit of storage.
BGP Role model: tracking the adoption of RFC 9234
Border Gateway Protocol route leaks occur when routing announcements propagate beyond their intended customer-provider or peer relationships, causing traffic misdirection and performance degradation. RFC 9234 embeds relationship awareness directly into BGP by establishing negotiated BGP Roles and attaching an Only to Customer (OTC) transitive path attribute to prevent unauthorized route propagation. To assess real-world adoption, Cloudflare monitored OTC attribute transmission across peer networks and discovered that 33.1 percent of IPv4 and 17 percent of IPv6 paths were missing the OTC attribute. Investigation revealed that Tier-1 transit providers AS1299 and AS3257 were stripping the attribute due to legacy defensive BGP error-handling configurations. Following direct engagement, AS1299 updated its configurations to preserve OTC propagation, whereas AS3257 continued stripping it, highlighting the necessity of broader vendor and carrier support across the Internet.
Evaluating AI Agents Live at the Grounded Reasoning Cup
The Grounded Reasoning Cup evaluated whether AI agents could generalize grounded reasoning improvements from OfficeQA to a newly released, enterprise-style document benchmark. Eleven academic teams had about two months to build agents with a partner lab’s model family, then applied them live to OfficeQA Pro V2, based on U.S. Treasury Accounts of Receipts and Expenditures, after a 36-hour release window. Stanford won with 63.3% accuracy, while the top systems combined document preprocessing, targeted or hybrid retrieval, specialized tools, parallelism, and answer verification; 18.8% of questions remained unsolved by every team. The results show that benchmark gains did not reliably transfer, and that parsing, retrieval, tool use, verification, and operational infrastructure can matter as much as model choice in end-to-end enterprise reasoning.
What's New with Monitoring in PostgreSQL 19
PostgreSQL 19, still in beta when this was written, adds monitoring and observability changes spanning lock contention, logging, WAL, wait events, remote messages, and multixacts. The default for log_lock_waits changes to on, while log_min_messages gains per-process-type levels and autoanalyze receives its own duration setting. A new wal_fpi_bytes counter exposes full-page-image volume cluster-wide, per session, per query, and in VACUUM and ANALYZE logs; new wait events expose WAL write, flush, and replay stages plus COPY file, pipe, and program I/O. Remote server messages now use normal PostgreSQL logging with origin prefixes, and pg_get_multixact_stats() surfaces multixact usage that previously required filesystem inspection and educated guessing. The post also lists upgrade work for parsers, metric collectors, wait-event dictionaries, configuration UIs, and alert thresholds, while noting that final release notes may differ.
Consistency is the new latency: AI at the data layer
Autonomous AI agents depend on the accuracy of the underlying data layer because retrieved data populates the LLM context window for multi-step reasoning. In distributed architectures, standard asynchronous replication lag causes agents to ingest stale data, leading to flawed execution and self-reinforcing hallucination debt when errors are saved back to memory. To resolve this, architectures must align replication strategies with specific task truth requirements across three primary patterns. High-stakes workloads utilize Amazon Aurora with global write forwarding or native Aurora DSQL for multi-region strong consistency, while high-scale scenarios leverage DynamoDB Global Tables with conditional writes to prevent lost updates. Finally, high-velocity ingestion pipelines enforce LOCAL_QUORUM reads and writes in Amazon Keyspaces to capture critical data spikes reliably.
How to Evaluate Production AI Agents: Measure System Outcomes, Not Conversations
Evaluating AI agents solely on conversational transcripts fails when agents must invoke tools to modify external production systems. Language models can convincingly narrate that an action occurred while completely omitting the underlying tool call, leaving database or billing records unchanged. Outcome-based evaluation addresses this divergence by treating external system mutations rather than generated dialogue as the actual source of truth. Frameworks such as CRMAgentBench implement persistent, stateful environments with strict all-or-nothing validation, verifying tool arguments, execution sequence, final record modifications, and the total absence of unauthorized actions. Furthermore, measuring repeated success across independent executions using pass^k metrics exposes operational variance and inconsistency that standard single-shot scores and pass@k metrics conceal.
No Dumb Questions: What is AI context architecture? Why not just build your own?
AI context architecture defines the boundaries, data access, memory, and fallback behavior that constrain AI agents and make their outputs more predictable. The discussion distinguishes architecture as system design, infrastructure as the storage and delivery of context, and engineering as implementation; MCP is treated as an architectural protocol, while RAG spans all three layers. Examples include restricting retrieval to car or sports-car tires, preserving agent history, and routing incomplete or low-trust knowledge to subject-matter experts through a human-in-the-loop flow. The buy-versus-build discussion notes that coding a custom architecture is possible, but buying can provide accumulated experience with data treatment, edge cases, and recurring operational problems, while good architecture also reduces token costs and aims for consistent, predictable answers.
Reading the signals: Helping your AI-built payment integration understand business intent
AI agents can quickly produce Stripe payment flows that create a PaymentIntent, render the Payment Element, and pass an end-to-end demo, yet a correct implementation can still violate the business intent behind the money. The post argues that payment expertise and business decisions are often missing from integrations built from API references, causing silent failures such as terminal declines being retried, intents stuck in requires_action, uncaptured authorizations, restricted connected accounts, and asynchronous payments being refunded. It recommends defining business intent before configuring the PaymentIntent, then writing those rules into code comments, metadata, documented requirements, or custom objects. A verifier agent can check rules on every build, either through an agent skill, automated tests, or CI, while shared ownership remains with the human who decides what correct means.
What are AI Hallucinations?
AI hallucinations are coherent, confident outputs that are factually wrong, fabricated, or unsupported, appearing in chatbots, image generators, and multimodal systems. Generative models predict likely continuations rather than retrieve guaranteed facts, while training incentives, limited or flawed data, generation settings, and error compounding in reasoning models can increase the risk. Examples include Google Bard’s incorrect JWST claim, Air Canada’s nonexistent bereavement policy, Microsoft Sydney’s erratic responses, and fabricated legal citations that led to sanctions. Hallucinations create enterprise risks involving safety, liability, compliance, security, reputation, and trust; recommended controls include guardrails, source verification, adversarial testing, monitoring, human review, retrieval-augmented generation, governed data, and evaluation throughout development and deployment.
Recovery strategies to meet data residency requirements
Organizations operating under data residency requirements must balance jurisdictional compliance with disaster recovery objectives for critical workloads on AWS. In locations served by a single Region or facing multi-Region constraints, three architectural patterns offer distinct trade-offs between regulatory adherence and operational complexity. The cryptographic boundary approach replicates encrypted data to another AWS Region while restricting decryption access using AWS Key Management Service policies. The data boundary pattern deploys AWS Outposts on-premises within the source country while relying on an external parent Region control plane. Finally, strict local autonomy boundaries retain data and control planes entirely in-country through on-premises systems or multi-cloud lifeboats, requiring rigorous end-to-end continuous testing.
Certificate Transparency Monitoring is now generally available
Cloudflare announced the general availability of Certificate Transparency Monitoring alongside a filtering mechanism to eliminate noisy alert emails. Previously, the monitoring system alerted domain owners for all Certificate Transparency log entries, including routine, automated renewals for Universal SSL, Advanced Certificate Manager, and backup certificates. Because the alerting flow and certificate ordering service operated independently, existing identifiers like TBSCertificate hashes arrived too late to verify pre-certificates. To resolve this race condition, the ordering service now computes an SHA-256 hash of the DER-encoded SubjectPublicKeyInfo structure at key generation. When the alerting service discovers a log entry, it recomputes this hash to verify ownership against the ordering database, suppressing alerts for Cloudflare-managed certificates while preserving alerts for external or unexpected issuances.
ClickStack and Hud bring runtime intelligence to AI-powered development
ClickHouse and Hud announce an integration that combines ClickStack’s service-level observability with Hud’s function-level runtime forensics for AI-assisted software development. As coding agents generate more code, the post identifies the central challenge as assessing production risk, detecting regressions, and giving engineers enough context to fix them. Teams connect the ClickStack and Hud MCP servers to a single coding agent, while shared trace IDs correlate OpenTelemetry traces and logs with Hud’s function-level metrics and forensic data; Hud’s Runtime Code Sensor requires no manual instrumentation and runs with low overhead in production. The resulting workflows support pre-deployment risk assessment against production behavior, release monitoring, auto-triage and auto-fix, and rollback or code-level remediation when issues arise.
Automated agent triage with Agent Tracing and Claude Routines
Evaluating Sentry's AI agent Seer previously required manually querying evaluation data and parsing raw JSON traces to find tool failures, latency spikes, and hallucinations. To automate this triage, the team deployed a daily Claude Routine connected to Sentry using a Model Context Protocol server. Each morning, the routine analyzes hundreds of overnight agent conversations, reviews aggregate error metrics, samples failed and successful conversation spans, and inspects the agent codebase for context. Claude evaluates whether final decisions match underlying reasoning and creates Linear tickets for newly discovered issues. This daily automated analysis accelerated the feedback loop, allowing engineers to identify efficiency issues like repeated malformed repository queries and deploy same-day fixes.
Databricks Network Configuration delivery to Tens of Millions of Serverless VMs
Databricks’ serverless platform launches tens of millions of VMs daily, and each VM needs network configuration before serving customer workloads. The original design synchronously aggregated data from multiple upstream services on the cluster-creation critical path, producing 5,000-ms p99 RPC latency and availability bottlenecks as request volume reached billions per day. The replacement uses change events through a message queue, asynchronous per-workspace recomputation, partition-local snapshots, and periodic reconciliation to recover from missed events. The serving path now performs a single snapshot-store read without upstream calls, while events carry workspace and resource identifiers rather than customer data. After rollout, p99 RPC latency fell to 125 ms, server success rate rose to 99.99%, upstream call volume dropped 86%, and the legacy synchronous framework was deprecated.
How to Choose Full-Stack Observability for NVIDIA AI Factories
AI infrastructure spans compute, networking, storage, orchestration, and applications, so a symptom at one layer may originate elsewhere. The post proposes selecting observability tools by failure domain, mapping components to telemetry sources, and reducing signals to a top-k alert set in a unified Prometheus/Grafana triage dashboard. For a DGX cluster using InfiniBand, BCM, and Slurm, it recommends Redfish/IPMI, DCGM, NVSM, UFM, and BCM, while deferring NetQ, NMX, Run:ai, and NIM until their corresponding capabilities are needed. The framework emphasizes DCGM for GPU metrics, NVSM for DGX system health, UFM for InfiniBand, and BCM for cluster and job aggregation, with specialized tools retaining low-level telemetry. An elevated InfiniBand bit error rate illustrates how retransmissions can slow one rank during NCCL all-reduce, causing synchronized training throughput to follow the slowest rank; alerts should connect concise signals to owners and remediation actions.
How we raised mobile end-to-end test stability to 98%
Shopify’s largest mobile app had become blocked by flaky end-to-end tests, which failed when screens took longer to render and had been removed from pull-request checks. Since 2023, tests used Appium through WebdriverIO and React Native Test IDs, allowing timing shortcuts and assertions about component-tree presence rather than what merchants could see or use. The rebuild hides Appium behind a strict builder-style wrapper that requires an assertion after every action; computer vision uses screenshots, PaddleOCR for text, and OpenCV for icons, with Test IDs as an opt-in fallback. After promotion into blocking CI, stability reached 98% of individual test successes over total runs, up from 50% with the old API, on two platforms. A pre-promotion flakiness gate rejects tests that exceed a failure threshold across repeated runs, while remaining failures largely involve network problems or simulator boot failures.
Taking AUTO CDC to the next level: Solving the hardest real-world use cases
The post describes new AUTO CDC capabilities for challenges that standard CDC patterns do not fully address, including out-of-order bitemporal timelines, partial updates, and durable audit history. Bitemporal AUTO CDC tracks business and system time independently through four managed interval columns, rewriting affected history when corrections arrive out of order so either point-in-time view can be reconstructed. Partial Updates are generally available and treat selected incoming NULL values as "do not update," preventing unchanged fields from overwriting stored values. The post also describes logging business and system timestamps in MLflow for reproducibility after VACUUM, and notes that the Python API for AUTO CDC Type 1 is being contributed to Apache Spark 4.2, with a SQL interface already merged for a next release.
What's new in pg_clickhouse v0.10.0: Subqueries, TPC-H Speedups, C Driver, and Aggregates
pg_clickhouse v0.10.0 expands pushdown for analytic workloads, with full pushdown across the TPC-H suite remaining the stated target. The release moves the scoreboard from 12 of 22 fully pushed-down queries to 16, including Q17, which falls from 32.7 seconds to 37 milliseconds when its correlated aggregate subquery is shipped to ClickHouse. SubPlan handling now turns PostgreSQL subqueries into ClickHouse subqueries, while NOT IN and related operators receive guards that preserve PostgreSQL’s three-valued NULL semantics; compatibility requires ClickHouse 25.8 or later for the correlated-subquery shape. The binary driver was rebuilt on a plain-C client library and hardened against concurrency bugs. The release also broadens pushed-down functions and aggregates, adds clickhouse_perform() and clickhouse_server_version(), and leaves six TPC-H queries plus other roadmap items unresolved.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.