---
title: "Reliability"
description: "289 posts about Reliability, summarised, each linking to the original."
---

# Reliability
> 289 posts about Reliability, summarised, each linking to the original.

## Articles

### [What does 99.9% uptime mean for inference?](https://yomu.fyi/post/what-does-99-9-uptime-mean-for-inference.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Jul 16, 2026
- Type: Explainer

The post defines three inference reliability tiers by the failure domains each must survive: 99% requires surviving node-level GPU failures like ECC errors and thermal throttling through automated health checks and fast replica replacement within a single data center. 99.9% requires surviving a full data-center failure by deploying weights across two facilities with enough capacity on each side and routing live traffic to both continuously, not a cold standby. 99.99% requires multi-region deployment with AZ redundancy and reserved failover capacity. The authors argue that vague SLA definitions hide which failure domains are covered, and that infrastructure ownership—whether a provider controls its hardware or rents from a hyperscaler—determines how quickly failures are resolved.


### [Security incident disclosure — July 2026](https://yomu.fyi/post/security-incident-disclosure-july-2026.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: system
- Published: Jul 16, 2026
- Type: Incident / postmortem

An intrusion into part of production infrastructure was driven end to end by an autonomous AI agent system, accessing a limited set of internal datasets and several credentials used by services. The attack began through two code-execution paths in dataset processing, then escalated from a worker to node-level access and moved laterally across internal clusters while running thousands of actions in short-lived sandboxes. AI-assisted anomaly detection surfaced the compromise, and LLM-driven analysis of more than 17,000 attacker events reconstructed the timeline, identified indicators, mapped credentials, and separated impact from decoys. Hosted frontier models initially blocked forensic requests containing attack artifacts, so the investigation used zai-org/GLM-5.2 on internal infrastructure. The initial vulnerabilities were closed, affected infrastructure rebuilt, credentials rotated, and cluster controls and alerting strengthened; assessment of possible partner or customer impact remains ongoing.


### [How Stripe uses graph search and state machines to auto-remediate a global database fleet](https://yomu.fyi/post/how-stripe-uses-graph-search-and-state-machines-to-auto-remediate-a-gl.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Pragya Mehta
- Published: Jul 16, 2026
- Type: Problem & solution

Stripe describes how its Document Database fleet previously handled degraded MongoDB shards with single-step, hard-coded plugins that struggled with combined failures, intermediate states, and more than 40 shard layouts. The replacement models shard topology and health as a directed graph, with infrastructure operations as edges and safety invariants defining valid states. A simulator uses breadth-first search to find minimum-operation recovery paths, while an abstraction over Temporal contexts lets the same remediation logic run in memory without real infrastructure calls; later, the system also uses Dijkstra's algorithm. The system reduced pager volume by 30%—about 200 pages annually—reclaimed 12 days of fleet health each year, and handled a new layout with zero code changes, according to the post.


### [From weeks to a day: how we made LLM evaluation fast enough to iterate on](https://yomu.fyi/post/from-weeks-to-a-day-how-we-made-llm-evaluation-fast-enough-to-iterate.md)
- Company: [Airbnb](https://yomu.fyi/company/airbnb.md)
- Author: Baharak Saberidokht
- Published: Jul 14, 2026

Airbnb built a four-layer LLM infrastructure framework to reduce iteration and evaluation turnaround from weeks to a single day. By using per-sample caching for generated references and judge scores, they established a deterministic evaluation foundation that separates genuine model drift from measurement noise. This setup enables rapid, bounded model hotfixes using micro LoRA adapters and comprehensive end-to-end validation across component boundaries.


### [Shipyard: How We Built Slack’s Next-Generation EC2 Platform](https://yomu.fyi/post/shipyard-how-we-built-slack-s-next-generation-ec2-platform.md)
- Company: [Slack](https://yomu.fyi/company/slack.md)
- Author: Archie Gunasekara
- Published: Jul 14, 2026
- Type: Problem & solution

Slack developed Shipyard to modernize its EC2 infrastructure by replacing continuously updated, mutable instances with immutable deployable artifacts. The platform adopts a layered image architecture starting with a standardized base image called slack-zero, which is built using AWS Image Builder and consumed by downstream service-specific machine images. Rather than running periodic Chef jobs in the background, configuration management is restricted to image baking and initial instance provisioning phases. Fleet management is handled through the Peekaboo inventory system, metric-driven progressive rollouts via Gondola, and automated instance rotation managed by a lifecycle component known as the Reaper. Developers validate cookbook updates prior to merging using Ship Quick, which provisions real infrastructure across dedicated worker fleets to verify provisioning workflows.


### [How we configure huge pages in ClickHouse Managed Postgres](https://yomu.fyi/post/how-we-configure-huge-pages-in-clickhouse-managed-postgres.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jul 14, 2026
- Type: Tutorial

ClickHouse Managed Postgres configures Linux huge pages to reduce the per-connection page-table cost and translation overhead of a large PostgreSQL shared\_buffers cache. The configuration reserves a quarter of machine memory before PostgreSQL starts, verifies HugePages\_Total, sets huge\_pages = 'on' to prevent silent fallback, and sizes shared\_buffers against the complete shared-memory segment, including buffer descriptors, WAL buffers, and lock tables. On an r7i.4xlarge with 32GB shared\_buffers, 4KB pages added 31.1MB of page tables per connection and reached 6.12GB at 200 connections, while 2MB huge pages reached 111MB. A select-only pgbench test at 100 clients also rose from 373,083 TPS to 418,087 TPS, a 12% increase, while startup failure without a sufficient pool made misconfiguration explicit.


### [Doing the right thing when things go wrong](https://yomu.fyi/post/doing-the-right-thing-when-things-go-wrong.md)
- Company: [Intercom](https://yomu.fyi/company/intercom.md)
- Author: Mark Gorman
- Published: Jul 14, 2026
- Type: Explainer

Fin’s incident-management process is designed to reduce customer impact quickly when deployments, alarms, or customer reports indicate a disruption. It combines customer and product signals, including heartbeat metrics for workflows such as Fin’s response generation, with engineer monitoring of rollouts; a rollback can land in a little under two minutes. Responses run through incident.io and Slack, with defined engineering lead, incident commander, business lead, and resolution owner roles, plus the out-of-hours STRIKE responder group. The lifecycle covers triage, investigation, mitigation, and learning, with a stated preference for rolling back to a known state before fixing forward and with explicit alternatives when rollback is unavailable. After recovery, bounded follow-ups and incident reviews aim to improve recurrence prevention, blast-radius reduction, detection, and response, while early escalation and clear communication make the process repeatable rather than dependent on heroics.


### [Enhancing Goodput in Large-Scale LLM Training with Nonuniform Tensor Parallelism](https://yomu.fyi/post/enhancing-goodput-in-large-scale-llm-training-with-nonuniform-tensor-p.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Michelle Horton
- Published: Jul 6, 2026
- Type: Problem & solution

Large-scale LLM training jobs spanning thousands of GPUs frequently encounter transient hardware interruptions that bottleneck tightly coupled tensor parallel groups. Nonuniform Tensor Parallelism addresses this bottleneck by dynamically adapting the tensor parallelism degree within affected scale-up domains rather than dropping entire data replicas. When a GPU drops out, the system reconfigures the remaining functional devices to process larger shards and resumes execution from the latest checkpoint. To prevent the downsized replica from stalling global synchronization, advanced rack designs temporarily power-boost the active GPUs to increase clock frequencies and match baseline throughput. Overlapping the necessary tensor resharding with backward computation and parameter synchronization reduces overhead on healthy replicas to under one percent.


### [Building Zone Failure Resilience in Apache Pinot™ at Uber](https://yomu.fyi/post/building-zone-failure-resilience-in-apache-pinottm-at-uber.md)
- Company: [Uber](https://yomu.fyi/company/uber.md)
- Published: Jul 2, 2026
- Type: Problem & solution

Uber describes how it made Apache Pinot clusters resilient to zone failures while preserving query and ingestion availability for Tier-0 use cases. It maps Uber’s virtual resource isolation group IDs to Pinot server pools, uses pool-based instance assignment with replica-group segment assignment, and requires at least as many isolation groups as the table’s replication factor. During migration, automated pool registration, configuration backfills, granular APIs, selective rebalancing, and rollback support were used across more than 400 Tier 0-1 clusters and thousands of tables. An internal drill found that one remaining replica continued serving traffic without query degradation after a zone went down, while isolation-group-based claim and release policies enabled parallel restarts and sequential rollouts with baking time.


### [Any Apple update can break our app. Here's how we find out first.](https://yomu.fyi/post/any-apple-update-can-break-our-app-here-s-how-we-find-out-first.md)
- Company: [Sentry](https://yomu.fyi/company/sentry.md)
- Author: Dan Mindru, Oleh Stasula
- Published: Jul 2, 2026
- Type: Problem & solution

Usage is an Apple ecosystem resource monitor that relies on undocumented APIs vulnerable to silent data structure changes across OS updates. To maintain stability across iPhone, iPad, and Mac without degrading device performance, the development team uses Sentry across separate component projects in a unified workspace. Beyond capturing standard crashes, the team wraps Swift errors with unexpected raw data payloads to quickly diagnose undocumented API structure drift while scrubbing sensitive user values. They also employ Sentry Size Analysis as a regression gate to monitor release payload deltas, preventing unintended asset omissions or bloat from slipping through code reviews. This multi-layered monitoring setup enables the team to detect and resolve platform-induced breakages before most end users encounter them.


### [Keeping funds where they belong: Implementing fund segregation in Stripe Connect](https://yomu.fyi/post/keeping-funds-where-they-belong-implementing-fund-segregation-in-strip.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Andrew Robinson
- Published: Jul 2, 2026
- Type: Explainer

Marketplaces using Stripe Connect’s separate charges and transfers can pool seller funds with platform revenue, exposing them to payouts, unrelated disputes, fees, refunds, and operational mistakes. In response to anticipated PSD3 changes and tighter Commercial Agent Exemptions, Stripe introduced Segregated Separate Charges and Transfers, which places captured payment funds in an allocated balance that cannot fund payouts and can only be transferred to connected accounts. The integration enables allocated\_funds on PaymentIntents, transfer\_group tracking, source\_transaction-linked transfers, and application fees collected from the same allocation; balances are inspected through the API. The post also details split transfers, refund and reversal ordering, dispute handling, prerequisites, and limitations including card-only support, preview-header requirements, and sandbox-only testing, presenting segregation as a way to make fund flows traceable and operationally safer.


### [The fuel of the future is already here: Why TRISO matters](https://yomu.fyi/post/the-fuel-of-the-future-is-already-here-why-triso-matters.md)
- Company: amazon.science
- Author: Katy Huff
- Published: Jun 24, 2026
- Type: Explainer

Amazon is investing in next-generation nuclear technology powered by tristructural isotropic (TRISO) fuel particles to satisfy the increasing energy demands of cloud computing and artificial intelligence infrastructure. Each sub-millimeter TRISO particle encloses a high-assay low-enriched uranium kernel within a porous carbon buffer and a three-layer ceramic shell featuring silicon carbide that acts as a robust miniature containment vessel. Testing by the U.S. Department of Energy demonstrated that irradiated particles withstand temperatures of 1600°C for 300 hours without detectable failures. In pebble-bed designs like the X-energy Xe-100 reactor, graphite spheres containing TRISO fuel circulate continuously through the core to enable online refueling, passive heat removal, and modular 80-megawatt deployments at facilities like the Cascade Energy Center.


### [Automated Schema Evolution in Pinterest’s Next-Generation DB Ingestion Framework](https://yomu.fyi/post/automated-schema-evolution-in-pinterest-s-next-generation-db-ingestion.md)
- Company: [Pinterest](https://yomu.fyi/company/pinterest.md)
- Author: Pinterest Engineering
- Published: Jun 24, 2026
- Type: Problem & solution

Pinterest's CDC ingestion platform relies on a multi-stage pipeline built with Kafka, Flink, Spark, and Iceberg, where upstream schema updates risk disrupting data transformations and offline storage. To prevent pipeline failures and schema drift, Pinterest introduced an automated schema evolution framework based on a staged convergence model. Upstream changes are detected via push-based DDL messages and daily pull-based validation checks. The workflow automatically restricts automated updates to additive changes and numeric precision widening, updating Iceberg table definitions and regenerating Flink and Spark code through auditable pull requests. By decoupling schema divergence, code deployment, and data convergence, the system maintains continuous ingestion availability while working toward eventual schema consistency across all storage layers.


### [Build Trust in Agentic AI: From POC to Production](https://yomu.fyi/post/build-trust-in-agentic-ai-from-poc-to-production.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Ken Wiebke
- Published: Jun 23, 2026
- Type: Explainer

Transitioning autonomous agentic AI systems from sandbox environments to enterprise production exposes a trust gap driven by hallucination, reasoning errors, and financial liability. To mitigate these risks, organizations can implement a proactive four-step framework covering foundations, verification, governance, and business outcomes. The architecture grounds agents using retrieval-augmented generation, short-term memory, long-term procedural memory, and observability traces consolidated on a unified data layer. Active verification computes an Agent Confidence Score alongside a deterministic Business Risk Score to produce an Agent Decision Score governing autonomy levels. Aggregated execution telemetry feeds business observability dashboards that monitor unit economics and enable analytical agents to recommend operational interventions.


### [What's New in pg\_clickhouse v0.3.2: Postgres 19, TLS, Regex, and Memory](https://yomu.fyi/post/what-s-new-in-pg-clickhouse-v0-3-2-postgres-19-tls-regex-and-memory.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 23, 2026
- Type: Announcement

pg\_clickhouse v0.3.2 updates the Postgres interface for querying ClickHouse, with changes centered on PostgreSQL 19 Beta1 support, TLS configuration, regular-expression pushdown, and memory consumption. The release adds CREATE SERVER options for forcing TLS or plaintext, selecting a minimum TLS version, and enabling ClickHouse native-protocol compression. Revised pushdown logic maps Postgres regular-expression flags to ClickHouse behavior, while regexp\_match() can now be pushed down when its pattern has no capturing groups. Two memory issues were fixed: one involving unbuffered HTTP-driver queries and another involving repeated foreign scans such as parameterized inner scans in nested-loop joins. The minor update requires no reload, restart, or ALTER EXTENSION UPDATE, and also fixes empty-array ANY() errors on ClickHouse versions before 25.


### [How Telcos Build Autonomous Networks with Agentic AI](https://yomu.fyi/post/how-telcos-build-autonomous-networks-with-agentic-ai.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Amogh Dendukuri
- Published: Jun 23, 2026
- Type: Explainer

Telecom operators currently operate mostly within Levels 2 and 3 of the TM Forum autonomous networks taxonomy, relying on predefined scripts. Progressing to Levels 4 and 5 demands autonomous systems capable of intent comprehension, real-time telemetry sensing, and cross-domain action coordination. To address this, an autonomy platform architecture combines domain-specific reasoning models, agent harnesses, and isolated sandboxed runtimes. Workflows separate responsibilities among on-demand agents, long-running execution loops, and deep-research multi-agent systems that evaluate ranked remediation paths or design wireless algorithms. Initial experiments in simulated SR-MPLS routing and wireless physical or MAC layer design show autonomous discovery can deliver measurable gains over standard operational and engineering baselines.


### [Inside NVIDIA Halos for Robotics: A Full-Stack Functional Safety System for Physical AI](https://yomu.fyi/post/inside-nvidia-halos-for-robotics-a-full-stack-functional-safety-system.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Suhas Hariharapura Sheshadri
- Published: Jun 22, 2026
- Type: Announcement

NVIDIA announced the launch of NVIDIA Halos for Robotics, a full-stack functional safety system designed for industrial robots, humanoids, and autonomous mobile robots. The platform extends NVIDIA's functional safety architecture from autonomous vehicles into robotics, utilizing NVIDIA IGX Thor and the NVIDIA Holoscan Sensor Bridge as its hardware foundation. On top of this hardware, Halos OS provides the base Halos Core safety operating system and application blueprints, including an Outside-In Safety Blueprint that leverages external worksite cameras and safety decision modules. Platform hardware safety features include an isolated IEC 61508 SIL 3 capable Safety Island alongside extensive fault-detection mechanisms. To accelerate compliance, the accredited NVIDIA Halos AI Systems Inspection Lab assesses partner integrations before final third-party certification.


### [The future of observability won’t be one proprietary AI agent. It will be thousands built by teams.](https://yomu.fyi/post/the-future-of-observability-won-t-be-one-proprietary-ai-agent-it-will.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 22, 2026
- Type: Explainer

Observability vendors are increasingly betting on a universal SRE agent that combines a platform, telemetry, and natural-language investigation. The post argues that debugging depends on team-specific systems, trusted signals, failure history, runbooks, workflows, and institutional knowledge that cannot be packaged into one vendor’s agent. Agents can compare more time windows and pursue more hypotheses than human investigators, creating demand for low-latency query infrastructure and more complete, high-fidelity historical data. It proposes a more open model in which teams build specialized agents on headless observability foundations, with APIs, data storage, and an observability MCP, while persistent investigation surfaces preserve work for collaboration and reuse. Humans remain responsible for direction, judgment, and action, so the near-term future is presented as human-agent collaboration rather than autonomous incident resolution.


### [How Spyne simplified their CDC pipeline with ClickPipes and ClickHouse Cloud](https://yomu.fyi/post/how-spyne-simplified-their-cdc-pipeline-with-clickpipes-and-clickhouse.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 22, 2026
- Type: Problem & solution

Spyne’s CDC pipeline streamed changes from MySQL and MongoDB through self-managed Debezium, Kafka, and ClickHouse, but table onboarding required 1 to 1.5 hours of manual work and schema drift could silently exclude new columns. The team migrated to ClickPipes and ClickHouse Cloud, where pipelines can be configured in minutes and automated schema evolution applies upstream changes to corresponding ClickHouse tables without manual intervention or restarts. To manage snapshot pauses, Spyne isolated about 10 critical historical tables in dedicated ClickPipes and grouped lower-priority tables separately. The migration also required using FINAL in downstream queries to handle ReplacingMergeTree eventual consistency, while binlog gaps may still require full table resynchronization and higher costs. Spyne reports less operational toil, faster feature and analysis workflows, and improved data consistency, accepting reduced recovery control and query-time CPU trade-offs.


### [The Data Canary: How Netflix Validates Catalog Metadata](https://yomu.fyi/post/the-data-canary-how-netflix-validates-catalog-metadata.md)
- Company: [Netflix](https://yomu.fyi/company/netflix.md)
- Author: Netflix Technology Blog
- Published: Jun 19, 2026
- Type: Problem & solution

A manual mitigation action during an incident corrupted a data feed for a subset of titles, causing playback issues and catalog service failures that existing code canary systems failed to catch. To protect streaming reliability, Netflix built an automated data canary system that validates transformed catalog metadata prior to publication. The architecture utilizes a dedicated orchestrator alongside permanent baseline and canary service clusters to coordinate validation using real production traffic. By leveraging custom chaos experiment thresholds, sticky session affinity, and Starts Per Second playback metrics, the system detects regressions in under ten minutes and blocks publication automatically. Controlled failure injection experiments routing approximately 0.2% of global traffic confirmed that issues could be identified in 2.5 to 4 minutes.


[Newer posts](https://yomu.fyi/topic/reliability/page/4.md) · [Older posts](https://yomu.fyi/topic/reliability/page/6.md)