Copy-and-Replay vs Branch-at-a-Point: Two Backup Recovery Paths
A side-by-side comparison of copy-and-replay recovery and branch-at-a-point restores, showing why restore time scales with data size in one model and not the other.
Loading…
Topic
Reliability
288 posts about Reliability. Every summary links to the original.
- 288 posts
- 46 companies
- 2 explainers
See the idea
Reliability explainers
The SQS Decision Most Diagrams Skip: Normal Work vs. Broken Work
Amazon SQS connects producers and workers, but its real design test is failure. Compare the processing path with the signals and controls needed when queue operations break.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Fast, fault-tolerant PyTorch training on AI Runtime
At scale, training jobs face frequent GPU interruptions, while remote-storage input pipelines can leave accelerators idle. Distributed Checkpoint (DCP) has each rank write a shard plus a .metadata file, allowing parallel saves and reloads onto a different number of GPUs; DCP is presented as useful for DDP as well as sharded training. Asynchronous saves stage data locally and upload in the background; cited measurements report 1.8x savings for a 2.8B-parameter DDP model and 58x for a 20B-parameter FSPD model on 32 H100s, though the comparison excludes torch.save network storage time. UCVolumeDataset and the accompanying DataLoader cache files on local NVMe and prefetch disjoint rank/worker slices while computation runs. Correct recovery also requires checkpointing data position plus seeds and RNG state so resumed sampling remains consistent.
LLMs Are Not (Consistently) Bayesian: Quantifying Internal (In)consistencies of LLMs’ Probabilistic Beliefs
LLMs are deployed in medicine, science, and law, where observed evidence may not determine a single correct answer and systems must represent and update uncertain beliefs. The work treats LLMs as information processing rules and measures an information processing gap, defined as deviation from Bayes updates, to quantify internal consistency in probabilistic belief updates. Extensive experiments compare multiple ways of incorporating evidence: some yield nearly Bayesian updates, while others rely on learned heuristics. Although exact Bayesian updating is optimal information processing, the non-Bayesian heuristic updates often achieve better downstream task performance, suggesting that the LLMs’ probabilistic models of the world are misspecified. The measure also serves as a diagnostic for issues in LLM-powered inferential systems.
Data Mesh at Grab (Part III): Operationalizing data reliability with automated DPIs
Grab’s data mesh uses the Signals Marketplace to publish, discover, and reuse trusted data products, while certification establishes ownership, documentation, lineage, and quality controls but does not prevent runtime failures. Data Production Issues (DPIs) turn data-quality signals into an operational workflow that begins when Kinabalu detects unhealthy contract tests and closes when the underlying condition recovers. Triage promotes meaningful contract breaches, deduplicates repeated signals, groups related issues, and records affected assets, intervals, contracts, tests, and dependencies. For diagnosis, the Data Health API exposes a portable taxonomy of UPSTREAM_ERROR, PLATFORM_ERROR, JOB_ERROR, and DATA_ERROR, allowing ownership to follow the likely root cause while platforms retain detailed internal diagnostics. Combined with Hugo’s diagnosis and remediation patterns, the approach is presented as reducing resolution time and feeding operational reliability back into certification.
Closing the AI agent trust gap with graduated autonomy
Operating AI agents under binary full-access or read-only permissions creates an operational trust gap due to unpredictable model behavior across updates. The graduated autonomy architectural pattern closes this gap by dynamically granting or revoking tool permissions through sustained reliability scoring. A scoring engine computes a composite score across five weighted dimensions over a rolling 50-action window, mapping agents across four tiers from probation to full autonomy. In-process pre-execution filters check for prompt injection and sensitive targets, while Amazon Bedrock AgentCore Gateway and Cedar policies strictly enforce deny-by-default access at the infrastructure level. Post-execution state capture in Amazon DynamoDB supports reversibility, and AWS CodePipeline delivery gates block releases failing adversarial evaluation benchmarks.
How Ubuntu Is Using Rust to Rebuild Core System Tools
Canonical is selectively adopting Rust to rewrite core system utilities across Ubuntu releases to improve memory safety, security, and long-term resilience. Starting in Ubuntu 26.04 LTS, uutils coreutils shipped as the default implementation targeting complete GNU compatibility, alongside sudo-rs, which introduced intentional behavior changes like asterisk display on password entry. Canonical is also funding ntpd-rs to consolidate NTP, NTS, and PTP into a single utility, while collaborating on UPKI to bring certificate revocation to system utilities. To manage Rust packaging at distribution scale, Canonical vendors dependencies per package rather than creating individual Debian packages for crates and embeds software bills of materials into binaries using cargo auditable. Future plans include exploring Rust implementations of compression libraries like bzip2-rs, zlib-rs, and zstd-rs.
Building a data plane from scratch: Stripe’s own high-performance distributed proxy
Stripe replaced Envoy in its service mesh after reliability, connection-cardinality, rollout-risk, and customization problems became increasingly difficult at its scale. Mesh-proxy, a Go proxy, implements HTTP/1.1 and HTTP/2, connects directly to Stripe’s service registry, and supports retries, hedging, rate limiting, and dynamic priority-based routing. Unlike Envoy’s worker-per-core model, it uses one read and one write goroutine per connection and multiplexes requests over a shared connection, reducing connection counts from one per worker to one per host. Load tests and deployment results showed about half the CPU under high load and roughly 50% lower latency, equivalent to around 3,000 days of cumulative inter-service latency saved daily. Direct registry integration also reduced unnecessary xDS updates, while custom Ruby balancing added 15–20% headroom before bursts overwhelm machines; Stripe is now pursuing anomaly detection and smart routing.
Restore LLM Inference Capacity in Seconds with Shadow Engine Recovery in NVIDIA Dynamo
LLM engine process failures typically trigger cold restarts that reload weights into HBM, compile kernels, and recapture CUDA graphs, leaving surviving workers to absorb displaced traffic for minutes. Shadow engine recovery, a preview feature in NVIDIA Dynamo, keeps a fully initialized standby on the same GPUs as the active engine. GPU Memory Service (GMS) preserves one physical weight copy across engine processes, while the parked shadow precomputes its CUDA context, graphs, communicators, and mappings but defers KV cache materialization. In a GLM-5.2 test, the shadow resumed service in 7.3 seconds after a worker failure, compared with 283 seconds for a cold restart, reducing post-fault p50 TTFT from 23,815 ms to 1,311 ms and improving decode rate from 12 to 46 tok/s/user. The preview does not cover hardware, node, or multi-node failures and currently starts promoted shadows with empty KV caches.
Read your writes: WAIT FOR in PostgreSQL 19
PostgreSQL 19 introduces the top-level SQL command WAIT FOR, addressing stale reads that can occur when an application writes to a primary and immediately reads from an asynchronously replicated standby. After the write, the client or pooler obtains an LSN with pg_current_wal_insert_lsn() and sends it to the reading session, where WAIT FOR blocks until the requested WAL position reaches the selected stage. The default standby_replay mode makes the change visible, while standby_write, standby_flush, primary_flush, TIMEOUT, and NO_THROW provide other durability and error-handling choices. The command waits without holding a snapshot, avoiding a replay self-deadlock, and exposes related wait events in pg_stat_activity. The feature preserves asynchronous writes by making only reads that require read-your-writes consistency wait, though timeline changes after promotion require caution.
ClickGap: Autonomous QA for ClickHouse
ClickGap addresses defects that survive ClickHouse’s extensive CI, human review, and in-house automated review, including untested behavior, silent resource waste, and performance regressions. The autonomous QA daemon examines eligible merged pull requests against verified builds, uses separate bug and coverage analysts, and requires hypotheses to become executed tests. An adversarial reviewer and ten evidence gates check consequences, source citations, complete-file review, reproducibility, and other conditions before a report can be filed. Five months after launch, ClickGap had filed roughly 500 issues and opened around 200 coverage or fix pull requests; more than half of the issues were closed, with over 200 linked to a fix PR. The system’s central design goal is to make reports trustworthy rather than merely prolific.
Transactional Vs Analytical Database: Choosing OLTP, OLAP, or Hybrid
Transactional and analytical databases serve different workloads: OLTP handles high-volume, real-time reads and writes with ACID guarantees, while OLAP supports complex queries over large historical datasets for reporting and business intelligence. The comparison focuses on trade-offs involving latency, concurrency, data freshness, schema design, transaction size, and storage format. Row-oriented storage and normalized schemas support precise operational updates, whereas column-oriented storage and denormalized designs improve compression, scans, and aggregations across millions of rows. The source describes transactional systems as foundational for applications such as banking, e-commerce, healthcare, and reservations, while analytical systems support dashboards, forecasting, and ad-hoc analysis. Its recommended architecture is to run both systems independently and connect them with Change Data Capture, balancing operational reliability with analytical throughput.
How a global payment processor preserved AWS RAM shares and Lake Formation permissions during an AWS Organizations migration
A global payment processor migrating 382 AWS accounts between AWS Organizations encountered control-plane failures when AWS Resource Access Manager automatically severed organization-bound resource shares. Although data planes continued operating for shared resources such as AWS Transit Gateways, infrastructure automation lost the ability to modify dependencies like AWS Glue Data Catalog databases. The engineering team resolved this by implementing temporary retained bridge shares, establishing external principal associations that survived cross-organization account transfers. Following each account move, automated tooling restored durable original shares, validated Lake Formation permissions, and deleted the bridge copies to prevent configuration drift. This bridge-share workflow enabled the migration of 378 accounts before a Transitional Service Agreement deadline without customer downtime.
Giga-Scale AI and the Ethernet Evolution: How Spectrum-X Ethernet Rewrites the Rules
Generative AI training across hundreds of thousands of GPUs makes the scale-out network a first-order performance bottleneck, because synchronized collectives expose weaknesses in conventional Ethernet. Static ECMP, packet loss controls such as PFC, and slowly tuned congestion control can create stragglers, head-of-line blocking, latency spikes, and multi-tenant interference. Spectrum-X Ethernet addresses these conditions with hardware-accelerated adaptive per-packet routing, targeted ECN- and RTT-based congestion control, and a SuperNIC Plane Load Balancer that distributes traffic across independent network planes. Spectrum-X Multiplane uses shallow two-tier fat trees and host-edge optical devices to provide path diversity while scaling a two-tier design to over 128,000 endpoints. In the cited tests, Spectrum-X sustained 98% of theoretical line rate across GPU pairs, held P99 tail latency to 8–9 microseconds under 75% load, recovered from a link flap in 2.68 milliseconds, and limited bandwidth loss under 10% fabric failure to 11%.
Ensuring reliable OpenTelemetry ingestion at scale
LogHouse’s OpenTelemetry pipeline ingests 50 million events per second and stores 177 PiB of uncompressed data, but its initial agent-to-gateway design could not reliably absorb ClickHouse backpressure. In-memory queues caused instability during short-lived spikes or database outages, while local collector write-ahead logs made gateways stateful, slowed recovery, delayed fresh telemetry behind FIFO backlogs, and took about four hours to drain 1 TiB. The team rejected adding Kafka because operating a new tier-zero service across cloud regions was not justified for this use case, then designed a custom architecture using blob storage, the OTel failover connector, overflow notifications, and a catchup collector. In staging, failover redirected 120,000 events per second to S3 without changing gateway resource usage, and recovery resumed ClickHouse writes while the catchup collector drained the queue; the design scaled from 10 million to 50 million events per second, with added regional setup and less precise recovery visibility.
Branded Login URLs and Email Workflows for Production Auth
Default authentication setups often redirect users to third-party domains and issue generic transactional emails, causing password manager failures, browser cookie restrictions, and perceived phishing risks. To mitigate these issues, teams can configure custom identity domains via DNS CNAME records to preserve origin alignment and session persistence. Developers can also capture custom user fields directly within hosted login prompts using Universal Login partials alongside pre-registration Actions rather than managing custom authentication servers. For transactional messaging, teams must replace rate-limited built-in email tools with dedicated external providers and configure SPF, DKIM, and DMARC records alongside Liquid-templated emails. Managing these configurations through version control and deployment tools maintains security standards without adding excessive infrastructure maintenance overhead.
amazon.science
SOP-Bench: A new benchmark for evaluating AI agents on real business procedures
AI agents often struggle to execute standard operating procedures because real-world business workflows contain implicit domain knowledge and procedural ambiguity that standard benchmarks overlook. To evaluate agent reliability, researchers introduced SOP-Bench, an open framework featuring more than 2,000 runnable tasks across 12 industrial domains paired with functioning tool interfaces and ground-truth answers. Domain experts authored the workflows, while an Anthropic Claude 3.5 Sonnet v2 model generated mock APIs, tool code, and edge-case datasets before expert verification. Experiments evaluating function-calling and reasoning agents across 11 frontier models revealed that newer models did not consistently outperform older versions, and providing 20 extraneous tools nearly halved task success. The benchmark, baseline agents, and evaluation code are released on GitHub and HuggingFace for reproducible testing.
What else runs on your Postgres server, and how do we stop it from taking the database down?
The Postgres VM described here runs PgBouncer instances, a backup agent, metrics exporters, a local Prometheus, a log collector, and watchdog timers alongside the database. Those processes share machine-wide resources outside Postgres’s huge-page-backed shared_buffers, so leaks, CPU saturation, or disk exhaustion can threaten database availability. ClickHouse Managed Postgres places supporting services in a cgroup v2 slice and combines Go runtime GOMEMLIMIT targets with systemd memory.high and memory.max enforcement. GOMEMLIMIT encourages earlier garbage collection, memory.high applies reclaim and throttling, and memory.max provides a hard ceiling whose OOM scope excludes Postgres because it uses a separate cgroup. CPU weights, bounded buffers, log and metric controls, disk thresholds, and exemptions for replication and monitoring sessions extend isolation across other failure modes; these controls are enabled by default.
Improved failure reports on Red Hat OpenShift with the event-driven diagnostic operator
During major infrastructure incidents, critical system logs and diagnostic evidence are frequently overwritten, lost, or erased by emergency recovery efforts before engineers can investigate. To solve this visibility gap in multi-cluster environments, an open-source event-driven diagnostic operator was developed as an automatic first information report collector. Deployed on a central hub cluster, the operator monitors Kubernetes warning events rather than relying on heavy polling. When an event matches a predefined regular expression pattern, the operator identifies the target spoke cluster, copies its kubeconfig secret, and spins up an independent must-gather job. Diagnostic data is written to off-site or shared storage with a configurable retention window, preserving forensic context without human intervention.
Where Security Fits in an AI Agent Stack
The post explains where security controls should sit in an emerging AI agent stack as agents gain capability and operate over longer horizons. It distinguishes behavioral controls in models and harnesses, which guide proposed actions, from authoritative infrastructure controls that bind identity, enforce policy, contain failures, and record activity. The stack is presented as layers for models, harnesses, meta-harnesses, secure runtimes such as NVIDIA OpenShell, and inference infrastructure, with the runtime boundary established before launch so tools, plugins, MCP processes, and delegated runtimes operate under policy. It recommends least privilege, isolation, explicit authorization, auditability, short-lived and task-scoped access, independent checks, and safer failure states, while describing four security profiles from isolated to adversarial workloads. The conclusion is that every high-impact external effect must pass through enforcement layers the agent cannot bypass.
How Hugging Face Inference Endpoints, Jobs, and Buckets Power Search on Papers with Code
Papers with Code needs a search engine that supports exact titles and arXiv identifiers, semantic requests, navigational queries, typos, and fast responses when model services are cold or unavailable. The system uses hybrid retrieval: PostgreSQL full-text search provides a lexical baseline, pgvector stores dense embeddings, and reciprocal rank fusion combines them. Full-corpus embeddings for more than 110,000 papers run in burstable GPU Jobs from repeatable-read database snapshots, with checksummed JSONL inputs and resumable Parquet outputs stored in Buckets. An Inference Endpoint serves low-latency query embeddings and bounded hourly updates for changed papers, while lexical retrieval remains a fallback. The embedding contract pins the model revision, dimension, prompts, normalization, and source hash; production uses Qwen/Qwen3-Embedding-0.6B with 256-dimensional L2-normalized vectors, and stored vectors also support related-paper retrieval without a request-time model call.
Securing Software at the Speed of AI
Palantir describes how its Product Security Team built an agentic software security strategy around multi-agent review, vulnerability hunting, triage, and runtime validation. The approach treats models as replaceable components within a governed harness that controls tools and data access, preserves evidence, coordinates specialized stages, independently challenges findings, and deduplicates reports using source-to-sink paths. Foundry supplies organizational context and lineage across code, services, findings, decisions, and remediation, while Apollo provides a controlled path for delivering verified fixes across heterogeneous software environments. The strategy positions remediation, rather than vulnerability identification, as the central bottleneck and extends these capabilities through Security Forge, a model-agnostic platform for customer cyber defense.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.