# Reliability > 61 posts about Reliability, summarised, each linking to the original. ## Articles ### [How Ubuntu Is Using Rust to Rebuild Core System Tools](https://yomu.fyi/post/how-ubuntu-is-using-rust-to-rebuild-core-system-tools.md) - Company: [Jetbrains](https://yomu.fyi/company/jetbrains.md) - Author: Irina Mihajlovic - Published: Aug 26, 2026 Canonical is selectively adopting Rust to rewrite core system utilities across Ubuntu releases to improve memory safety, security, and long-term resilience. Starting in Ubuntu 26.04 LTS, uutils coreutils shipped as the default implementation targeting complete GNU compatibility, alongside sudo-rs, which introduced intentional behavior changes like asterisk display on password entry. Canonical is also funding ntpd-rs to consolidate NTP, NTS, and PTP into a single utility, while collaborating on UPKI to bring certificate revocation to system utilities. To manage Rust packaging at distribution scale, Canonical vendors dependencies per package rather than creating individual Debian packages for crates and embeds software bills of materials into binaries using cargo auditable. Future plans include exploring Rust implementations of compression libraries like bzip2-rs, zlib-rs, and zstd-rs. ### [MetaRoCE: A New RDMA Transport Built for AI-Scale Ethernet](https://yomu.fyi/post/metaroce-a-new-rdma-transport-built-for-ai-scale-ethernet.md) - Company: [Meta](https://yomu.fyi/company/meta.md) - Author: By Arvind Srinivasan, Neil Spring, Omar Baldonado, Rajiv Krishnamurthy - Published: Aug 24, 2026 Scaling frontier AI training and inference across massive GPU clusters places severe demands on network fabrics, yet conventional RoCE relies on Priority Flow Control to enforce in-order packet delivery. Meta introduced MetaRoCE, a clean-sheet RDMA transport protocol engineered specifically for lossy, multiplane commodity Ethernet without requiring PFC. By shifting intelligence to endpoint NICs, the protocol sprays packets across fine-grained logical paths, writing out-of-order data directly into destination memory without reorder buffers or head-of-line blocking. Congestion is managed via per-path ECN windows paired with receiver-driven rate hints returned in acknowledgments. In validation on a 64-node AMD GPU cluster running collective operations, MetaRoCE outperformed RoCEv2, sustaining roughly 86% throughput under 1% packet loss and achieving linear scaling across multiplane fabrics. ### [How we knew COVID was over (and what our models had to unlearn)](https://yomu.fyi/post/how-we-knew-covid-was-over-and-what-our-models-had-to-unlearn.md) - Company: [Airbnb](https://yomu.fyi/company/airbnb.md) - Author: Harrison Katz - Published: Aug 19, 2026 Airbnb's forecasting team outlines their framework for managing production models that experience drift or external shocks. Rather than treating model updates as a generic retraining task, they distinguish between refitting with new data, respecifying the underlying model structure, and intentionally holding without changes. This discipline prevents teams from chasing temporary noise, carrying obsolete crisis assumptions, or over-engineering solutions during disruptions. ### [Improving infrastructure efficiency for growing demand in the age of AI](https://yomu.fyi/post/improving-infrastructure-efficiency-for-growing-demand-in-the-age-of-a.md) - Company: [Dropbox](https://yomu.fyi/company/dropbox.md) - Author: Ilya Yakovlev - Published: Aug 18, 2026 Dropbox manages growing infrastructure demands from AI and standard workloads by maximizing the efficiency of its existing data centers rather than just building more capacity. Through a system-level approach spanning software, hardware, and physical facility design, the company dynamically manages power and balances workloads. These optimizations, combined with higher storage density and data-driven hardware lifecycle management, have significantly reduced energy use per unit of storage. ### [BGP Role model: tracking the adoption of RFC 9234](https://yomu.fyi/post/bgp-role-model-tracking-the-adoption-of-rfc-9234.md) - Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md) - Author: Bryton Herdes - Published: Aug 18, 2026 Border Gateway Protocol route leaks occur when routing announcements propagate beyond their intended customer-provider or peer relationships, causing traffic misdirection and performance degradation. RFC 9234 embeds relationship awareness directly into BGP by establishing negotiated BGP Roles and attaching an Only to Customer (OTC) transitive path attribute to prevent unauthorized route propagation. To assess real-world adoption, Cloudflare monitored OTC attribute transmission across peer networks and discovered that 33.1 percent of IPv4 and 17 percent of IPv6 paths were missing the OTC attribute. Investigation revealed that Tier-1 transit providers AS1299 and AS3257 were stripping the attribute due to legacy defensive BGP error-handling configurations. Following direct engagement, AS1299 updated its configurations to preserve OTC propagation, whereas AS3257 continued stripping it, highlighting the necessity of broader vendor and carrier support across the Internet. ### [How to Evaluate Production AI Agents: Measure System Outcomes, Not Conversations](https://yomu.fyi/post/how-to-evaluate-production-ai-agents-measure-system-outcomes-not-conve.md) - Company: [Salesforce](https://yomu.fyi/company/salesforce.md) - Author: Scott Nyberg - Published: Aug 17, 2026 Evaluating AI agents solely on conversational transcripts fails when agents must invoke tools to modify external production systems. Language models can convincingly narrate that an action occurred while completely omitting the underlying tool call, leaving database or billing records unchanged. Outcome-based evaluation addresses this divergence by treating external system mutations rather than generated dialogue as the actual source of truth. Frameworks such as CRMAgentBench implement persistent, stateful environments with strict all-or-nothing validation, verifying tool arguments, execution sequence, final record modifications, and the total absence of unauthorized actions. Furthermore, measuring repeated success across independent executions using pass^k metrics exposes operational variance and inconsistency that standard single-shot scores and pass@k metrics conceal. ### [Certificate Transparency Monitoring is now generally available](https://yomu.fyi/post/certificate-transparency-monitoring-is-now-generally-available.md) - Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md) - Author: Jenny Yang - Published: Aug 13, 2026 Cloudflare announced the general availability of Certificate Transparency Monitoring alongside a filtering mechanism to eliminate noisy alert emails. Previously, the monitoring system alerted domain owners for all Certificate Transparency log entries, including routine, automated renewals for Universal SSL, Advanced Certificate Manager, and backup certificates. Because the alerting flow and certificate ordering service operated independently, existing identifiers like TBSCertificate hashes arrived too late to verify pre-certificates. To resolve this race condition, the ordering service now computes an SHA-256 hash of the DER-encoded SubjectPublicKeyInfo structure at key generation. When the alerting service discovers a log entry, it recomputes this hash to verify ownership against the ordering database, suppressing alerts for Cloudflare-managed certificates while preserving alerts for external or unexpected issuances. ### [Grab Bench: Evaluating AI on Grab-shaped production work](https://yomu.fyi/post/grab-bench-evaluating-ai-on-grab-shaped-production-work.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Christian Coffrant - Published: Aug 12, 2026 AI models evaluated on internal tasks often produce subtly plausible errors, such as altering metric definitions in SQL queries or missing hidden stateful invariants in code patches. To address this, Grab built Grab Bench, a configurable evaluation harness that executes model providers through specialized task plugins. The platform evaluates synthetic or redacted datasets that preserve production constraints across domains like query generation, tool use, passenger-profile reasoning, and agentic coding. By recording granular row-level outputs with deterministic scorers and LLM judges, the harness exposes failure modes rather than relying solely on aggregate leaderboards. ### [Cloudflare DDoS Threat Report H1 2026: 1 Tbps attacks soar as DNS floods and geopolitical tensions drive a new wave](https://yomu.fyi/post/cloudflare-ddos-threat-report-h1-2026-1-tbps-attacks-soar-as-dns-flood.md) - Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md) - Author: Cloudforce One - Published: Aug 11, 2026 In the first half of 2026, network-layer DDoS attacks exceeding 1 Tbps surged by 519% quarter-over-quarter, with 935 mitigated attacks recorded across the period. Threat actors shifted their focus toward reflection and amplification methods, making DNS-based floods and CLDAP floods the dominant vectors. Despite the rise in hyper-volumetric traffic, 96.62% of network-layer attacks remained under 500 Mbps and 90.60% ended in under 10 minutes. Geopolitical conflicts and international events heavily influenced target selection, pushing the Media, Production & Publishing industry to the top position. Because brief multi-gigabit bursts trigger downstream routing instability and application timeouts before analysts can respond, automated, always-on edge mitigation has become critical. ### [Unveiling good and bad behaviors on the Agentic Internet](https://yomu.fyi/post/unveiling-good-and-bad-behaviors-on-the-agentic-internet.md) - Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md) - Author: Jin-Hee Lee - Published: Aug 7, 2026 Web traffic increasingly shifts between human interaction and automated agents, rendering point-in-time security checks ineffective against sophisticated automation. Cloudflare addresses this shift through continuous behavioral analysis designed to evaluate trust across entire user sessions rather than relying on static point-in-time hurdles. Its CDN-injected client-side system, Precursor, tracks continuous behavioral indicators like cursor movements to catch mid-session automated takeovers. To counter rapid bot adaptation, Cloudflare is introducing self-adjusting machine learning models under Adaptive Intelligence and non-deterministic defenses such as the AI Labyrinth. These tools allow site owners to poison scraper training data, trap crawlers in synthetic webs, or queue legitimate agents while dynamically penalizing abusive traffic. ### [How Salesforce Eliminated Single-Region Risk and Reduced Downtime Blast Radius at 4B Metrics/Min](https://yomu.fyi/post/how-salesforce-eliminated-single-region-risk-and-reduced-downtime-blas.md) - Company: [Salesforce](https://yomu.fyi/company/salesforce.md) - Author: Scott Nyberg - Published: Aug 5, 2026 Salesforce redesigned its internal observability platform, Argus, to eliminate single-region failure risks while ingesting approximately 4 billion metrics per minute. Operating out of a single AWS region previously created a global blast radius and generated high cross-region data transfer costs. To mitigate these risks without the prohibitive expense of full multi-region replication, the engineering team implemented a geo-local architecture that processes and stores telemetry closer to its origin across production geographies. A newly introduced federation query layer relies on Elasticsearch mappings to route queries selectively to relevant regional clusters, avoiding broad fan-out. The platform also adopted metadata caching for wildcard queries and implemented HTTP 206 partial-response handling to maintain visibility when individual regions experience downtime. ### [How AI is transforming analytics at Grab](https://yomu.fyi/post/how-ai-is-transforming-analytics-at-grab.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Maanas Prabhakar - Published: Aug 1, 2026 Grab is restructuring its analytics operations using a five-level AI autonomy ladder, transitioning analysts from manual artifact creation to problem framing and decision governance. The architecture leverages domain-specific systems like Spartan to process natural language queries through certified metric indexes and Scarlet to triage and repair failing data pipelines. To prevent agent hallucinations, ContextIQ manages context lifecycles by automatically updating metric definitions, SQL references, and golden-dataset test cases when instrumentation changes or failures occur. Furthermore, data teams use an internal portal called BriX to configure custom analytics surfaces and automated root-cause analysis commentaries using reusable Model Context Protocol connections. Autonomy scales mechanical query and validation tasks while keeping human oversight focused on canonical metric definitions and strategic sign-offs. ### [Building Reliable Production AI with Durable Workflows](https://yomu.fyi/post/building-reliable-production-ai-with-durable-workflows.md) - Company: [Salesforce](https://yomu.fyi/company/salesforce.md) - Author: Scott Nyberg - Published: Jul 27, 2026 AI prototypes rely on simple prompt-response interactions, but production AI systems handling thousands of records must manage distributed execution issues like worker crashes, rate limits, and transient failures. Rather than treating long-running operations as single large jobs or rebuilding state from logs, systems require durable workflows that preserve execution history across failures. In Agentforce Grid, execution is organized into a hierarchy where column runs divide into recoverable batches and rows composed of discrete activities. Each batch checkpoints after every successful row, retrying up to ten times with exponential backoff and resuming strictly at incomplete work to prevent duplicated model calls. Exposing progress across these hierarchical boundaries ensures that both users and engineers can distinguish completed, failing, and retryable units. ### [From weeks to a day: how we made LLM evaluation fast enough to iterate on](https://yomu.fyi/post/from-weeks-to-a-day-how-we-made-llm-evaluation-fast-enough-to-iterate.md) - Company: [Airbnb](https://yomu.fyi/company/airbnb.md) - Author: Baharak Saberidokht - Published: Jul 14, 2026 Airbnb built a four-layer LLM infrastructure framework to reduce iteration and evaluation turnaround from weeks to a single day. By using per-sample caching for generated references and judge scores, they established a deterministic evaluation foundation that separates genuine model drift from measurement noise. This setup enables rapid, bounded model hotfixes using micro LoRA adapters and comprehensive end-to-end validation across component boundaries. ### [Shipyard: How We Built Slack’s Next-Generation EC2 Platform](https://yomu.fyi/post/shipyard-how-we-built-slack-s-next-generation-ec2-platform.md) - Company: [Slack](https://yomu.fyi/company/slack.md) - Author: Archie Gunasekara - Published: Jul 14, 2026 Slack developed Shipyard to modernize its EC2 infrastructure by replacing continuously updated, mutable instances with immutable deployable artifacts. The platform adopts a layered image architecture starting with a standardized base image called slack-zero, which is built using AWS Image Builder and consumed by downstream service-specific machine images. Rather than running periodic Chef jobs in the background, configuration management is restricted to image baking and initial instance provisioning phases. Fleet management is handled through the Peekaboo inventory system, metric-driven progressive rollouts via Gondola, and automated instance rotation managed by a lifecycle component known as the Reaper. Developers validate cookbook updates prior to merging using Ship Quick, which provisions real infrastructure across dedicated worker fleets to verify provisioning workflows. ### [The Data Canary: How Netflix Validates Catalog Metadata](https://yomu.fyi/post/the-data-canary-how-netflix-validates-catalog-metadata.md) - Company: [Netflix](https://yomu.fyi/company/netflix.md) - Author: Netflix Technology Blog - Published: Jun 19, 2026 A manual mitigation action during an incident corrupted a data feed for a subset of titles, causing playback issues and catalog service failures that existing code canary systems failed to catch. To protect streaming reliability, Netflix built an automated data canary system that validates transformed catalog metadata prior to publication. The architecture utilizes a dedicated orchestrator alongside permanent baseline and canary service clusters to coordinate validation using real production traffic. By leveraging custom chaos experiment thresholds, sticky session affinity, and Starts Per Second playback metrics, the system detects regressions in under ten minutes and blocks publication automatically. Controlled failure injection experiments routing approximately 0.2% of global traffic confirmed that issues could be identified in 2.5 to 4 minutes. ### [Sitar-agent: Building a reliable dynamic configuration sidecar at scale](https://yomu.fyi/post/sitar-agent-building-a-reliable-dynamic-configuration-sidecar-at-scale.md) - Company: [Airbnb](https://yomu.fyi/company/airbnb.md) - Author: Bo Teng - Published: Jun 4, 2026 Airbnb modernized sitar-agent, a Kubernetes sidecar that delivers dynamic configuration updates to thousands of polyglot service instances without requiring redeployments. The architecture uses periodic AWS S3 snapshots to enable fast, decoupled pod startups and maintains local file-based storage for the main container to read configs with in-memory caching. Key design decisions include retaining an isolated sidecar over an in-process library and optimizing pull-based polling with server-side caching. ### [Slack AI: The Path to Multi-Cloud](https://yomu.fyi/post/slack-ai-the-path-to-multi-cloud.md) - Company: [Slack](https://yomu.fyi/company/slack.md) - Author: Shaurya Kethireddy - Published: May 28, 2026 Slack evolved its Slack AI serving infrastructure across multiple phases to handle enterprise LLM workloads reliably and securely. The initial deployment on AWS SageMaker provided zero-knowledge escrow VPC isolation and FedRAMP compliance, but engineers faced scaling latency, GPU scarcity, and significant operational overhead. Slack then migrated live traffic to Amazon Bedrock to leverage managed Model Units and eliminate model release lag without customer-facing incidents. However, fixed Provisioned Throughput commitments and regional peak traffic variations created persistent underutilization challenges. Consequently, Slack expanded into a multi-cloud orchestration architecture that normalizes disparate provider APIs, integrates unified cross-cloud telemetry, and routes traffic dynamically around latency spikes and outages. ### [Introducing Nova, our internal platform for coding agents](https://yomu.fyi/post/introducing-nova-our-internal-platform-for-coding-agents.md) - Company: [Dropbox](https://yomu.fyi/company/dropbox.md) - Author: Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura,Mike White,Kevin Altschuler - Published: May 21, 2026 Dropbox developed Nova, an internal platform that runs AI coding agents in isolated cloud environments integrated with their Bazel monorepo. The platform supports both interactive developer workflows and autonomous background tasks, such as automated CI debugging, flaky test remediation, and codebase-wide migrations. By pairing code generation with automated validation and strict execution guardrails, Nova ensures generated fixes are tested and reproducible. ### [Enhancing Flink deployment with shadow testing](https://yomu.fyi/post/enhancing-flink-deployment-with-shadow-testing.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Tee Long Lang - Published: May 7, 2026 Production deployments of Apache Flink applications at Grab previously risked up to ten minutes of downtime when new versions failed under live traffic or could not resume from existing checkpoints. To eliminate these deployment failures, Grab's Coban data streaming team introduced an optional shadow testing stage directly into their production Kubernetes deployment pipeline. The shadow pipeline runs new application releases in parallel with the active application for a default observation period of one hour before prompting promotion to main. Dynamic runtime configuration based on an injected environment variable redirects source consumer groups and isolates output to dedicated shadow sinks, metrics, and logs. [Older posts](https://yomu.fyi/topic/reliability/page/2.md)