---
title: "Reliability"
description: "289 posts about Reliability, summarised, each linking to the original."
---

# Reliability
> 289 posts about Reliability, summarised, each linking to the original.

## Articles

### [Dispatches from O'Reilly: From capabilities to responsibilities](https://yomu.fyi/post/dispatches-from-o-reilly-from-capabilities-to-responsibilities.md)
- Company: [Stack Overflow](https://yomu.fyi/company/stack-overflow.md)
- Author: Artur Huk
- Published: Jun 19, 2026
- Type: Problem & solution

High-stakes AI agents capable of mutating external state often face governance failures when relying on system prompts or manual Human-in-the-Loop approval queues that quickly degrade into alert fatigue. The Responsibility-Oriented Agent architecture addresses this operational bottleneck by shifting system design from open-ended capability framing to deterministic, contract-enforced responsibilities. Under this model, underlying orchestration frameworks like LangChain operate in User Space with their side-effecting tools removed, isolating the agent to epistemic reasoning. The agent expresses its intended action exclusively by emitting a structured policy proposal to a privileged Kernel Space runtime. The runtime deterministically evaluates the proposal against versioned YAML contracts registered in an agent registry, ensuring that only genuine policy exceptions are escalated to human supervisors.


### [Introducing Postgres to Postgres ClickPipes in ClickHouse Cloud](https://yomu.fyi/post/introducing-postgres-to-postgres-clickpipes-in-clickhouse-cloud.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 11, 2026
- Type: Announcement

Postgres-to-Postgres migrations between providers can require manual schema handling, data movement, CDC, and cutover coordination, with risks around database objects and long-running loads. Postgres to Postgres ClickPipes, now in public beta, provides a fully managed migration service in ClickHouse Cloud designed to reduce operational overhead and support minimal downtime. It automates schema dump and restore, including secure connections, role conflicts, version differences, transactional application, retries, and UI or notification-based error reporting, while PeerDB supplies parallel initial loads and CDC capabilities. Independent pull and push processes use retries and alerting for transient errors, and the roadmap includes features to assist with cutover into Postgres by ClickHouse.


### [EC2’s formally verified “isolation engine” provides mathematical assurance of virtual-machine isolation](https://yomu.fyi/post/ec2-8217-s-formally-verified-8220-isolation-engine-8221-provides-mathe.md)
- Company: amazon.science
- Author: Dominic Mulligan, Nathan Chong
- Published: Jun 10, 2026
- Type: Announcement

AWS has introduced M9g and M9gd EC2 instances powered by Graviton5 CPUs, featuring the formally verified Nitro Isolation Engine to enforce virtual machine isolation. To prove the engine's correctness, engineers developed 330,000 lines of machine-checked mathematics using the Isabelle/HOL proof assistant. The system architecture separates hypervisor policy from isolation mechanisms, requiring the deprivileged Nitro Hypervisor to route all guest state modifications through the minimal Rust-based engine. Functional verification uses a restricted language subset called micro Rust, Separation Logic contracts specifying total correctness, and weakest-precondition calculus from the open-source AutoCorrode library. Finally, high-level transition relations and noninterference proofs ensure that memory integrity and guest confidentiality are preserved across every execution step.


### [Real-world grounding in agentic AI](https://yomu.fyi/post/real-world-grounding-in-agentic-ai.md)
- Company: amazon.science
- Author: Rose Yu
- Published: Jun 8, 2026
- Type: Problem & solution

Foundation models deployed as physical AI agents face critical risks when natural language hallucinations translate into physical law violations and operational hazards. To ensure physical consistency and reliability, researchers from the University of California, San Diego and Amazon Fulfillment Technology introduced four grounding pillars. Physics-guided deep learning embeds symmetries and differential equations into pretraining to obey conservation laws while reducing dataset requirements. The UQ4CT framework applies mixture-of-experts architectures to produce calibrated uncertainty estimates, cutting expected calibration error by over 25% and halting operations when safety thresholds are breached. Furthermore, the Adapting-While-Learning framework and formal verifiers like Hilbert integrate numerical simulators and interactive proof systems to enforce mathematical and scientific correctness.


### [Managing Elasticsearch Reindex at Scale: Performance, Reliability, and Observability](https://yomu.fyi/post/managing-elasticsearch-reindex-at-scale-performance-reliability-and-ob.md)
- Company: [Palantir](https://yomu.fyi/company/palantir.md)
- Author: Palantir
- Published: Jun 8, 2026
- Type: Explainer

Palantir’s document store needs to repair or rebuild Elasticsearch indices when mappings, settings, cluster versions, or index state change, while continuing to serve live search and writes. Its reindex machinery reconstructs schema-aware indices from the strongly consistent database, rather than relying on Elasticsearch’s \_reindex API, and is designed to limit manual work and resource consumption across many deployments and schemas. For an online reindex, it creates a physical shadow index with the intended mappings and settings, backfills it from the database, dual-writes current API changes to both indices, and promotes the shadow index after completion; searches remain on the primary during the rebuild. Cross-cluster migrations use a separate machine with cluster-specific metadata, configuration, index and alias evolution, and preservation of old-cluster indices to support rollback.


### [Errors, traces, logs, metrics: when to reach for what](https://yomu.fyi/post/errors-traces-logs-metrics-when-to-reach-for-what.md)
- Company: [Sentry](https://yomu.fyi/company/sentry.md)
- Author: Sergiy Dybskiy
- Published: Jun 5, 2026
- Type: Explainer

Modern application telemetry relies on four overlapping signals: errors, traces, logs, and metrics. Each telemetry type addresses distinct engineering questions and powers specific debugging workflows. Errors track grouped, actionable application crashes, whereas traces map timing and execution paths across dependency waterfalls. Metrics aggregate numerical measurements across historical deployments, and logs preserve complete state and decision-making context at specific moments in execution. Emitting telemetry directly into the format needed for a given workflow proves superior to relying on a single wide event, as tools like AI coding agents require structured spans to optimize execution paths while human debuggers rely on unsampled logs and metrics to pinpoint silent failures.


### [Sitar-agent: Building a reliable dynamic configuration sidecar at scale](https://yomu.fyi/post/sitar-agent-building-a-reliable-dynamic-configuration-sidecar-at-scale.md)
- Company: [Airbnb](https://yomu.fyi/company/airbnb.md)
- Author: Bo Teng
- Published: Jun 4, 2026

Airbnb modernized sitar-agent, a Kubernetes sidecar that delivers dynamic configuration updates to thousands of polyglot service instances without requiring redeployments. The architecture uses periodic AWS S3 snapshots to enable fast, decoupled pod startups and maintains local file-based storage for the main container to read configs with in-memory caching. Key design decisions include retaining an isolated sidecar over an in-process library and optimizing pull-based polling with server-side caching.


### [Multigres v0.1 Alpha: an operating system for Postgres](https://yomu.fyi/post/multigres-v0-1-alpha-an-operating-system-for-postgres.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: sugu\_sougoumarane
- Published: Jun 4, 2026
- Type: Announcement

Multigres v0.1 alpha is an open-source release of a system intended to provide Vitess-grade horizontal scaling, high availability, and operational simplicity for Postgres. It manages Postgres instances with sharding, connection pooling, automatic failover, and backup orchestration, while this release adds advanced pooling, automatic failovers, and a Kubernetes operator. Its high-availability protocol builds on unmodified Postgres replication, supports user-defined durability policies, resolves split-brain scenarios without losing succeeded commits, and permits replica changes while running. The release is suitable for experimentation but not production workloads: v0.1 is single-shard, sharding is absent, the custom-resource API may change, backward compatibility is not guaranteed, and performance benchmarks are still in progress.


### [Direct Preference Optimization Beyond Chatbots](https://yomu.fyi/post/direct-preference-optimization-beyond-chatbots.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Erick Lachmann, Gabriel Pimenta de Freitas Cardoso, Francisco de Almeida Rocha Alves
- Published: Jun 3, 2026
- Type: Explainer

DharmaOCR applies Direct Preference Optimization (DPO) beyond chat alignment to reduce text degeneration in structured OCR, where outputs are either acceptable transcriptions or failure loops. In a benchmark of Brazilian Portuguese structured-document OCR, vanilla degeneration rates among open-source model families ranged from below 1% to above 33%, while supervised fine-tuning (SFT) did not consistently reach production-acceptable levels. After SFT, the pipeline generated multiple candidates for 23,726 documents, used an automated LLM judge to score them, and retained the model’s degenerate outputs as rejected examples paired with chosen outputs. DPO reduced degeneration in every tested family, by 59.4% on average and up to 87.6%, without reducing extraction quality according to the benchmark; applicability is presented as dependent on failures being categorically identifiable, reliably scoreable, and sufficiently numerous.


### [Enterprise Business Software and the Mixed-Up Chameleon Problem](https://yomu.fyi/post/enterprise-business-software-and-the-mixed-up-chameleon-problem.md)
- Company: [Palantir](https://yomu.fyi/company/palantir.md)
- Author: Palantir
- Published: May 29, 2026
- Type: Explainer

Traditional ERP practice prioritizes a standardized, stable core, but the post argues that forcing every process into it can erode an organization’s identity and differentiated ways of creating value. It proposes keeping the ERP as the system of record while placing an interoperability layer above it. This layer uses a shared ontology to reconcile concepts across systems and supports workflows, automation, user experiences, agentic AI, and human engineering under shared security and governance. Because these capabilities remain outside the core, organizations can preserve distinctive processes during ERP upgrades and treat migration as a continuous software- and ontology-driven activity rather than a one-time manual exercise. The intended model is structured adaptability: standardized ERP functions retain discipline, auditability, and financial integrity while operational software adapts to the business.


### [Compliance versus conversion: Tax ID validation at checkout](https://yomu.fyi/post/compliance-versus-conversion-tax-id-validation-at-checkout.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Jake Estes
- Published: May 29, 2026
- Type: Explainer

Tax ID collection at checkout supports compliant tax invoices and tax determination, including reverse-charge treatment for qualifying cross-border business transactions. Validation can be performed synchronously before purchase completion or asynchronously after checkout: synchronous checks reduce correction and audit risk but may increase abandonment and expose conversion to government-service downtime, while asynchronous checks preserve completion but require follow-up on failures. The implementation discussion covers direct national-registry integrations, third-party validation services, and Stripe’s CheckoutSessions and PaymentIntents integrations, where the Tax ID Element uses verification modes of if\_supported or none. Stripe first applies a country-specific format check, routes validly formatted IDs to government APIs, may fall back to asynchronous processing after four seconds, and exposes status through customer.tax\_id.updated and the customer object.


### [Slack AI: The Path to Multi-Cloud](https://yomu.fyi/post/slack-ai-the-path-to-multi-cloud.md)
- Company: [Slack](https://yomu.fyi/company/slack.md)
- Author: Shaurya Kethireddy
- Published: May 28, 2026
- Type: Problem & solution

Slack evolved its Slack AI serving infrastructure across multiple phases to handle enterprise LLM workloads reliably and securely. The initial deployment on AWS SageMaker provided zero-knowledge escrow VPC isolation and FedRAMP compliance, but engineers faced scaling latency, GPU scarcity, and significant operational overhead. Slack then migrated live traffic to Amazon Bedrock to leverage managed Model Units and eliminate model release lag without customer-facing incidents. However, fixed Provisioned Throughput commitments and regional peak traffic variations created persistent underutilization challenges. Consequently, Slack expanded into a multi-cloud orchestration architecture that normalizes disparate provider APIs, integrates unified cross-cloud telemetry, and routes traffic dynamically around latency spikes and outages.


### [How flat is replacing fat in AWS data center networks](https://yomu.fyi/post/how-flat-is-replacing-fat-in-aws-data-center-networks.md)
- Company: amazon.science
- Author: Giacomo Bernardi, Ratul Mahajan, Seshadhri Comandur
- Published: May 28, 2026
- Type: Problem & solution

Traditional data center networks rely on hierarchical fat-tree topologies, which introduce overhead, upper-layer congestion, and vulnerability to router failures. Although flat random networks theoretically maximize route diversity and resilience, routing computation and cabling complexity historically made them impractical for physical data centers. AWS addressed these constraints by introducing Resilient Network Graphs, a quasi-random architecture powered by a passive optical device called ShuffleBox and a routing algorithm named Spraypoint. Spraypoint sprays packets across random neighbors toward designated waypoints, unlocking twice as many independent paths while operating within commodity router memory limits. Adopted as the default architecture across new AWS facilities, this design reduces router counts by 69%, lowers projected network power consumption by 40%, and increases throughput by up to 33%.


### [Under the River](https://yomu.fyi/post/under-the-river.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: May 28, 2026
- Type: Explainer

Shopify describes how its AI agent, River, grew from an infrastructure bet aimed at making a large, fragmented codebase usable by AI. In early 2024, the company adopted a monorepo called World and Nix as a reproducible substrate for development, CI, and production, despite substantial migration costs. River operates in public Slack channels, where it reads code, runs tests, queries data, inspects production traces, and opens pull requests; in a recent 30-day period it handled 59,918 sessions and contributed to 3,536 merged pull requests. Underneath it, Aquifer provides durable sessions, sandboxes, gateways, event logs, credentials handling, and observability, allowing agent profiles to share one platform across interactive, automated, and batch modes. The central design principle is that the session must survive process and machine failure, while public transcripts create a compounding knowledge corpus.


### [How the lakebase architecture stays resilient to cloud failures](https://yomu.fyi/post/how-the-lakebase-architecture-stays-resilient-to-cloud-failures.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Jasraj Dange, Hans Norheim
- Published: May 27, 2026
- Type: Explainer

Lakebase is adapting its reliability architecture to agentic and on-demand workloads, which create far more control-plane activity, database starts, and pressure on finite cloud capacity. Lakebase uses stateless Postgres compute over zone-resilient storage, dedicated multi-availability-zone computes for optional high availability, and a vertically autoscaling virtualization layer that reduces dependence on cloud provisioning control planes. A dedicated data-plane controller is being developed for hot-path start and suspend operations, while cell-based regional deployments isolate failures and allow capacity to expand incrementally. The reliability program also uses component and whole-availability-zone failure injection, consistency checks, Service Level Indicators, and Service Level Objectives, including startup, failover, storage, and API measurements. The stated target is for every database to exceed 99.99% monthly availability; reported 2026 attainment through April shows 99.75% to 99.85% of databases meeting that threshold.


### [Modern Java at Stripe: Language upgrades as a service](https://yomu.fyi/post/modern-java-at-stripe-language-upgrades-as-a-service.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Mahib Hosain
- Published: May 27, 2026
- Type: Problem & solution

Large JVM codebases often stall Java upgrades because build-graph constraints require coordinated changes across many owners, while manual scripts and stale inventories repeatedly rediscover blockers. Stripe built AutoJDK to continuously analyze the live build graph, compute eligibility using reverse-dependency, runtime, framework, and dependency-ordering constraints, and generate bounded, idempotent pull requests rather than opaque rewrites. It also tracks adoption and blocker categories, uses AI assistance for selected remediation tasks, and can operate continuously or on specified packages and services. Centralized JDK and runtime policy, CI checks, multi-JDK support, static analysis, and packaging applications with platform-specific JDKs support gradual transitions. After three months, the upgrader had merged more than 160 PRs, made JDK 17 the most common target, eliminated Java 8 services, and reduced the JDK 8 and 11 footprint by roughly 25K targets.


### [Protecting your Supabase projects from npm supply chain attacks](https://yomu.fyi/post/protecting-your-supabase-projects-from-npm-supply-chain-attacks.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: katerina\_skroumpelou
- Published: May 26, 2026
- Type: Explainer

The post explains why npm supply-chain attacks matter to Supabase projects: Edge Functions, the Supabase CLI, and packages including supabase-js, @supabase/ssr, and @supabase/server all depend on npm. Attackers can compromise maintainers, register typosquats, or exploit build pipelines; once malicious code reaches node\_modules, lifecycle scripts may read environment variables, cloud metadata, tokens, kubeconfig, and SSH keys. The proposed defense is layered: upgrade to pnpm 11, increase minimumReleaseAge to several days, pin sensitive dependency versions, commit and review lockfiles, disable or allowlist install scripts, verify package names, and pin GitHub Actions to full commit SHAs. Supabase is publishing a canonical security guide, hardening its GitHub Actions, adding secret-handling warnings to API documentation, and coordinating communications; the post also recommends rotating credentials after a potentially compromised install and using scanners as supplementary defense.


### [Using observability data to prevent incidents](https://yomu.fyi/post/using-observability-data-to-prevent-incidents.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Madelyn Mullen
- Published: May 22, 2026
- Type: Problem & solution

Engineering teams often remain reactive not because they lack telemetry, but because metrics, logs, traces, error budgets, and SLO burn rates are difficult to query at decision speed. The post frames reliability intelligence as trend analysis that identifies accumulating risk before a user-facing incident, contrasting it with observability focused on current system state. It proposes unified telemetry access and natural-language self-service, using Databricks Genie to examine p99 latency, error-budget burn, deployment frequency, incident rates, dependencies, and capacity across services. The Trade Desk’s Genie Router handles questions across more than 10,000 tables, while the post says ad-hoc analysis runs five times faster with Genie and can support mitigation before a page fires.


### [Introducing Nova, our internal platform for coding agents](https://yomu.fyi/post/introducing-nova-our-internal-platform-for-coding-agents.md)
- Company: [Dropbox](https://yomu.fyi/company/dropbox.md)
- Author: Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura,Mike White,Kevin Altschuler
- Published: May 21, 2026

Dropbox developed Nova, an internal platform that runs AI coding agents in isolated cloud environments integrated with their Bazel monorepo. The platform supports both interactive developer workflows and autonomous background tasks, such as automated CI debugging, flaky test remediation, and codebase-wide migrations. By pairing code generation with automated validation and strict execution guardrails, Nova ensures generated fixes are tested and reproducible.


### [The product analytics you already have](https://yomu.fyi/post/the-product-analytics-you-already-have.md)
- Company: [Sentry](https://yomu.fyi/company/sentry.md)
- Author: Rahul Chhabria
- Published: May 21, 2026
- Type: Explainer

Software engineers frequently rely on separate product analytics tools and teams to answer user behavior questions, creating disjointed data models and delayed insights. Existing debugging telemetry within tools like Sentry—specifically spans, structured logs, and application metrics—can serve as the primary source for product metrics. Developers enrich request spans with business-level attributes, record discrete business events through wide structured logs, and track KPIs with metric counters and distributions. This unified telemetry links business outcomes directly to underlying traces, errors, releases, and source code lines, enabling immediate root-cause analysis and automated alerts. While multi-session cohort and funnel aggregation remains a current tooling gap, standard telemetry solves day-to-day adoption, performance, and revenue questions.


[Newer posts](https://yomu.fyi/topic/reliability/page/5.md) · [Older posts](https://yomu.fyi/topic/reliability/page/7.md)