---
title: "Reliability"
description: "289 posts about Reliability, summarised, each linking to the original."
---

# Reliability
> 289 posts about Reliability, summarised, each linking to the original.

## Articles

### [Coding Agent Horror Stories: The Security Crisis Threatening Developer Infrastructure](https://yomu.fyi/post/coding-agent-horror-stories-the-security-crisis-threatening-developer.md)
- Company: [Docker](https://yomu.fyi/company/docker.md)
- Author: Jennifer Kohl
- Published: May 18, 2026
- Type: Problem & solution

Autonomous AI coding agents increasingly handle developer workflows, but their execution model introduces critical security risks. Operating with the executing user's full permissions, these agents can read ambient secrets, inherit cloud credentials, and run destructive shell commands without human intervention. Documented incidents between October 2024 and February 2026 reveal systemic hazards, including unintentional home directory deletions and increased security vulnerabilities in generated code. To address these vulnerabilities, Docker Sandboxes isolate agent execution within dedicated microVMs, preventing access to the host filesystem and blocking common credential paths by default. Routing changes through Git worktrees and restricting network egress ensures that agent actions remain strictly bounded and reviewable before affecting main branches or production infrastructure.


### [Operator: A look under the hood](https://yomu.fyi/post/operator-a-look-under-the-hood.md)
- Company: [Intercom](https://yomu.fyi/company/intercom.md)
- Author: Jack Ryan
- Published: May 15, 2026
- Type: Explainer

Operator is a customer-operations Agent built to help teams understand, manage, and improve customer experience across workspaces. The post distinguishes a production Agent from a prototype that simply gives an LLM API access and a system prompt, describing more than 50 purpose-built tools and 10 skills for data retrieval, reporting, investigation, and operational tasks. Its intelligence layer combines semantic search tuned against millions of support conversations, workspace-specific attribute awareness, and encoded reasoning, while the action layer supports configuration and content changes through reviewable diffs requiring explicit approval. A hybrid conversational and graphical interface presents charts, dashboards, and structured proposals within the existing platform. The post argues that read-only custom tools can be built in weeks, but production reliability, safe actions, multi-tenant isolation, graceful degradation, and ongoing maintenance require substantially greater investment.


### [We replaced Redis with MySQL for inventory reservations—and it scaled](https://yomu.fyi/post/we-replaced-redis-with-mysql-for-inventory-reservations-and-it-scaled.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: May 12, 2026
- Type: Problem & solution

Shopify replaced its Redis-based inventory reservation system with MySQL to unify reservations with the inventory ledger and provide ACID coordination between payment and inventory updates. The new design stores one row per sellable unit in a bounded pool of up to 1,000 rows per item/location combination, uses MySQL 8's SKIP LOCKED to avoid waiting on locked units, and replenishes depleted pools. Composite primary keys reduced row locks, READ COMMITTED avoided gap locks that interfered with replenishment, and consistent lock ordering prevented deadlocks. The investigation found connection-pool usage elsewhere in checkout, rather than reservation query speed alone, was the immediate ceiling; cleanup removed 50% of primary-database reads and 33% of transactions, while configuration changes added headroom. Shopify validated the design in shadow mode, then rolled it out gradually with a Redis rollback path, meeting high-throughput targets during peak 2025 traffic.


### [vLLM V0 to V1: Correctness Before Corrections in RL](https://yomu.fyi/post/vllm-v0-to-v1-correctness-before-corrections-in-rl.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Rafael Pardinas, Ehsan Kamalloo
- Published: May 6, 2026
- Type: Problem & solution

PipelineRL migrated its rollout inference engine from vLLM V0 (0.8.5) to V1 (0.18.1), but trainer-side logprobs, policy ratios, KL, entropy, clip rate, and reward diverged from the V0 reference. The investigation separated semantic mismatch, inference-path mismatch, and objective mismatch, then prioritized backend parity before changing the RL objective. Fixes included processed rollout logprobs, explicit V1 runtime settings disabling prefix caching and async scheduling, a V0-like inflight weight-update path, and an fp32 lm\_head for the final projection. With those changes, the final V1 run tracked the V0 trajectory across the reported metrics, while processed\_logprobs alone did not resolve the training gap.


### [Navigating uncertainty in Amazon's middle-mile network](https://yomu.fyi/post/navigating-uncertainty-in-amazon-apos-s-middle-mile-network.md)
- Company: amazon.science
- Author: Ruth Misener, Hana Ku, Georgios Paschos
- Published: May 6, 2026
- Type: Problem & solution

Amazon's middle-mile logistics network coordinates tens of millions of packages across hundreds of facilities while managing subtle demand variations and major operational disruptions. To solve this mixed-integer optimization challenge without combinatorial explosion, Amazon models routes at coarse time resolutions augmented by precomputed 15-minute timing bounds. The system uses Monte Carlo methods alongside a graph attention network featuring interconnected site graphs and origin-destination graphs to capture complex spatial correlations and flow interdependencies. By generating synthetic flow permutations, planners stress-test candidate network designs against hundreds of plausible scenarios to prioritize routing optionality. This risk-aware modeling approach preserves network flexibility, absorbs severe shocks, and yields potential cost savings while safeguarding delivery promises.


### [Introducing Application Metrics: Track the signal, see the spike, jump to the trace](https://yomu.fyi/post/introducing-application-metrics-track-the-signal-see-the-spike-jump-to.md)
- Company: [Sentry](https://yomu.fyi/company/sentry.md)
- Author: Ben Coe
- Published: May 5, 2026
- Type: Announcement

Sentry launched Application Metrics to capture application signals without pre-aggregating counters or discarding high-cardinality attributes like user IDs, project IDs, and regions. The system supports counters, distributions, and gauges within the existing SDK while linking each metric event directly to trace IDs for integrated debugging. Engineers demonstrated the feature by troubleshooting a Session Replay failure that triggered silently when sessions exceeded 1,000 video segments. By instrumenting distributions for replay event counts with project and replay attributes, the team isolated seven affected occurrences, navigated to connected traces and log lines, and reproduced the issue. Application Metrics comes enabled in recent SDK versions with 5GB included per plan.


### [How GitHub uses eBPF to improve deployment safety](https://yomu.fyi/post/how-github-uses-ebpf-to-improve-deployment-safety.md)
- Company: [Github](https://yomu.fyi/company/github.md)
- Author: Lawrence Gripper
- Published: Apr 16, 2026
- Type: Problem & solution

Deployment scripts can introduce dangerous circular dependencies when they rely on services or assets from platforms that are currently experiencing outages. Blocking network access at the host level is impractical because stateful nodes continue serving live traffic during rolling deployments. To solve this, GitHub isolates deploy scripts into dedicated Linux cGroups and attaches custom eBPF programs via the cilium/ebpf Go library. The system uses socket-address hooks to redirect DNS queries to a userspace proxy that checks a domain blocklist, while egress packet hooks map DNS transaction IDs to process IDs. This approach successfully prevents deploy-time circular dependencies, provides full command-line audit logs for blocked requests, and speeds up incident recovery.


### [How we built Stripe Credits: A programmable, auditable way to pay your Stripe fees](https://yomu.fyi/post/how-we-built-stripe-credits-a-programmable-auditable-way-to-pay-your-s.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Pratik Gupta
- Published: Apr 16, 2026
- Type: Explainer

Stripe Fee Credits replaces ad-hoc discounts and manual cash adjustments with a programmable, auditable virtual balance dedicated to eligible Stripe fees. The architecture has three layers: a standalone gRPC credits engine for metadata, policies, lifecycle, and audit history; logical financial accounts built on Money Movement and Storage; and fee-settlement orchestration that applies eligible credits before the primary balance. Logical accounts carry ledger balances without representing real cash, while platform checks restrict Originated Money Transfers to logical-to-logical movements and exclude these accounts from cash sweeps. Prepaid and Stripe-funded credits share the user experience but use separate funds flows: prepayments create deferred revenue and collect tax at drawdown, whereas Stripe-funded credits are booked as Stripe costs with taxes written off. The resulting infrastructure supports incentives, prepaid annual fees, and other programs across 26+ countries, with Dashboard balances and real-time auditability, while avoiding bespoke engineering for each commercial model.


### [We're Transferring the Stripe Sync Engine to Stripe](https://yomu.fyi/post/we-re-transferring-the-stripe-sync-engine-to-stripe.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: raminder\_singh, matt\_linkous, gregor\_vand, ana\_mogul
- Published: Apr 14, 2026
- Type: Announcement

The Stripe Sync Engine repository is moving from supabase/stripe-sync-engine to stripe/sync-engine after growing from an open-source tool for syncing Stripe data into a Stripe schema in Postgres. The transfer follows collaboration that added cursor-based incremental sync, JSONB storage with generated columns, multi-account support for Stripe Connect platforms, event ordering in a new CLI, and automatic incremental backfilling using Supabase Cron and Queue. The latest release adds coupon synchronization, branching support, immediate syncing on installation, one-click upgrades, SSL enforcement, and tighter installation controls, along with clearer errors and dashboard visibility for Edge Function source code. The project remains open source under Apache 2.0, existing installations continue to work, and the Supabase dashboard integration is unchanged. With Stripe data in Postgres, users can join it with application data for subscription checks, feature gating, MRR, and churn analysis.


### [Managing context in long-run agentic applications](https://yomu.fyi/post/managing-context-in-long-run-agentic-applications.md)
- Company: [Slack](https://yomu.fyi/company/slack.md)
- Author: Dominic Marks
- Published: Apr 13, 2026
- Type: Problem & solution

Long-running multi-agent systems struggle with context management because accumulating raw message histories degrades inference quality, increases latency, and exceeds context window limits. In a collaborative security investigation platform, passing unrestricted history can also introduce confirmation bias across specialized agents. To maintain coherence across unbounded rounds of investigation, the system eliminates raw message history carryover between invocations. Instead, it coordinates agents through three structured context channels: a Director's Journal for orchestration memory, a Critic's Review that scores findings to filter hallucinations, and a Critic's Timeline of validated chronological events. This architecture provides agents with tailored context without overwhelming their inference capacity.


### [Re-imagining ML Serving Infra: My Winter Internship at Ramp](https://yomu.fyi/post/re-imagining-ml-serving-infra-my-winter-internship-at-ramp.md)
- Company: [Ramp](https://yomu.fyi/company/ramp.md)
- Author: Michael Jiang
- Published: Apr 10, 2026
- Type: Problem & solution

Ramp’s ML Platform team needed to separate flexible model development in ml-pipelines from the Intelligence Platform’s 99.9%+ uptime and low-latency requirements. The internship project, Bring-Your-Own-Model-Weight Serving (mw-serve), built a dedicated serving layer that pulls model weights from MLflow and a training-environment base image from ECR, then runs each model in an isolated instance. To protect uptime, it added deployment safeguards, pre-traffic liveness checks, and monitors, while latency work instrumented client TCP/TLS connections and server processing. The investigation traced extreme tail latency to connection churn, which improved with longer keepalives and startup warmups, and to garbage-collection pauses associated with memory drops; tuning thresholds and freezing objects removed more outliers. By the final day, one Intelligence Platform model was serving production traffic through mw-serve, with additional models in shadow testing.


### [Selective Test Execution at Stripe: Fast CI for a 50M-line Ruby monorepo](https://yomu.fyi/post/selective-test-execution-at-stripe-fast-ci-for-a-50m-line-ruby-monorep.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Aditya Anchuri
- Published: Apr 9, 2026
- Type: Problem & solution

Stripe’s Ruby monorepo contains about 50 million lines, 100,000 test files, and roughly 1.2 million test units, making sequential execution impractical at an estimated four months. To keep CI within minutes, Selective Test Execution (STE) runs tests whose recorded file accesses intersect files changed since a suitable baseline, averaging about 5% of the suite and under 10% of always-run compute. A dynamically linked C++ library loaded with LD\_PRELOAD intercepts file opens, associates them with hierarchical test scopes, and propagates tracing to child processes; logs are aggregated into a selection index outside the syscall hot path. Guardrails cover tests affected by directory discovery, previously failing tests, selective linting, generated files, and reproducible database-backed baselines ordered by Monotonic Revision IDs. The system is presented as a way to preserve safety and affordability while scaling CI for a highly dynamic language.


### [How Eisan made POS analytics faster, cheaper, and more reliable with ClickHouse Cloud](https://yomu.fyi/post/how-eisan-made-pos-analytics-faster-cheaper-and-more-reliable-with-cli.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse
- Published: Apr 8, 2026
- Type: Problem & solution

Eisan System Development moved a retail analytics workload from a QlikView-based platform to ClickHouse Cloud after rising ID-POS data volumes caused slower processing, failed analyses, and increasing licensing and server costs. Client records from supermarkets and drugstores support basket analysis, cross analysis, and real-time personalization, with some datasets nearing 200 million records annually. The team compared open-source options including PostgreSQL and YugabyteDB, then tested ClickHouse configurations, including a 30-core, 120-GB setup, while accounting for cache memory usage through load testing. In comparison, ClickHouse ran on slightly lower-spec hardware under high-load conditions without observed performance deterioration, and Eisan reports no cases so far where it failed to return a result; the team is now targeting 100 billion records and exploring off-hours processing and scheduled scaling.


### [Zero downtime Upgrade: Yelp’s Cassandra 4.x Upgrade Story](https://yomu.fyi/post/zero-downtime-upgrade-yelp-s-cassandra-4-x-upgrade-story.md)
- Company: [Yelp](https://yomu.fyi/company/yelp.md)
- Author: Mark Surnin and Muhammad Junaid Muzammil, Software Engineer
- Published: Apr 7, 2026
- Type: Problem & solution

Yelp's Database Reliability Engineering team upgraded over a thousand Cassandra nodes from version 3.11 to 4.1 without downtime. The upgrade delivered reliability, observability, and performance improvements, including Java 11 support, configurable guardrails, and fixed incremental repairs. To avoid the high costs, consistency compromises, and data streaming delays of standing up separate datacenters, the team selected an in-place rolling upgrade strategy. The rollout orchestrated staged node updates alongside compatible versions of the Stargate proxy and custom data pipeline connectors without requiring client code modifications. Although Stargate 2.x introduced regressions on range and multi-partition queries that required downgrading to 1.x, the finished migration achieved up to a 58% reduction in p99 latency across key clusters.


### [Frontend Engineering at Palantir: Building a Backend-less Cross-Application API](https://yomu.fyi/post/frontend-engineering-at-palantir-building-a-backend-less-cross-applica.md)
- Company: [Palantir](https://yomu.fyi/company/palantir.md)
- Author: Palantir
- Published: Apr 6, 2026
- Type: Explainer

Palantir’s Application Frameworks team builds the “backend of the frontend”: frameworks that let applications communicate during workflows spanning multiple interfaces. The work addresses environments with denied, disrupted, intermittent, or limited (DDIL) connectivity, where relying on backend services can introduce unreliable latency. The team generalized the Gaia Controller prototype into Shared State, which lets applications define named data models and exchange updates organized by Media Type, and Commands, which exposes arbitrary logic through command IDs and request objects that return responses. Both frameworks use frontend-only browser APIs for client-side interaction, support workflows such as pairing Gaia with Workshop, and are also being used as tools by AIP agents; the post says adoption is growing across Palantir.


### [Improving storage efficiency in Magic Pocket, our immutable blob store](https://yomu.fyi/post/improving-storage-efficiency-in-magic-pocket-our-immutable-blob-store.md)
- Company: [Dropbox](https://yomu.fyi/company/dropbox.md)
- Author: Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura,Mike White,Kevin Altschuler,Facundo Agriel
- Published: Apr 2, 2026

After introducing a direct erasure-coding service, Dropbox encountered severe storage fragmentation in its immutable blob store, Magic Pocket, due to a large number of under-filled volumes. Because their legacy compaction strategy was only designed to top off near-full volumes, the team created new multi-strategy compaction methods to aggressively consolidate sparse volumes. These approaches leverage dynamic programming and streaming pipelines to rapidly reclaim unused disk capacity at exabyte scale.


### [From Custom to Open: Scalable Network Probing and HTTP/3 Readiness with Prometheus](https://yomu.fyi/post/from-custom-to-open-scalable-network-probing-and-http-3-readiness-with.md)
- Company: [Slack](https://yomu.fyi/company/slack.md)
- Author: Carlo Preciado
- Published: Mar 31, 2026
- Type: Problem & solution

Slack encountered a lack of client-side observability when introducing HTTP/3 at its edge. Because HTTP/3 runs over QUIC on UDP instead of TCP, neither existing commercial SaaS tools nor the internal Prometheus Blackbox Exporter could probe the new endpoints. To address this gap, an engineering intern integrated the open-source quic-go library into Prometheus Blackbox Exporter to build a native, configurable HTTP/3 probe. Slack deployed an internal implementation using the upstream functionality while waiting for pull request reviews, ultimately landing the contribution upstream. The resulting setup provides a unified view of HTTP/1.1, HTTP/2, and HTTP/3 metrics within Grafana, enabling reliable alerting and faster correlation with other telemetry.


### [Boost Training Goodput: How Continuous Checkpointing Optimizes Reliability in Orbax and MaxText](https://yomu.fyi/post/boost-training-goodput-how-continuous-checkpointing-optimizes-reliabil.md)
- Company: [Google](https://yomu.fyi/company/google.md)
- Author: Shutong Li, Colin Gaffney
- Published: Mar 31, 2026
- Type: Benchmark

Orbax and MaxText introduce continuous checkpointing to balance training reliability and performance when hardware failures or preemption events interrupt long-running jobs. Fixed checkpoint intervals create a trade-off: infrequent saves risk substantial lost work, while frequent saves can block or bottleneck training, particularly over unstable networks. MaxText enables asynchronous checkpointing that attempts a new save only after the previous background request is fulfilled; its continuous mode overrides checkpoint\_period, and max\_num\_checkpoints\_to\_keep can limit storage use. A llama-3.1-70B benchmark on two v5p-128 slices found smaller P50 checkpoint intervals, but average training step time increased because of more frequent device-to-host transfers; the analysis estimated resource conservation against mean-time-between-failure assumptions. Orbax also supports minimum intervals and customizable preservation policies, while co-locating storage with training is recommended because cross-metro bandwidth can reduce checkpointing speed and reliability.


### [Navigating Regional Network Blocks](https://yomu.fyi/post/navigating-regional-network-blocks.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: sara\_read
- Published: Mar 26, 2026
- Type: Incident / postmortem

Supabase describes how regional network blocks disrupted access to its multi-tenant developer platform and shares three incidents from September 2025 through March 2026. It explains that DNS-level or transport-layer blocking can affect every tenant when projects share domains and IP addresses, while adding a hosting domain to the Public Suffix List provides only partial mitigation. In the UAE, a project distributing unauthorized streaming content triggered an approximately 18-day block that ended after removal and engagement with the TDRA. Yemen saw TCP connections dropped for about eight days with no official explanation, while India’s roughly eight-day restriction was rescinded after project enforcement and technical engagement with MeitY. The post says providers should build regulator contacts, strengthen abuse response pipelines, and develop alternatives such as proxy domains, custom domains, and other workarounds.


### [5 Career Myths From Women in Engineering](https://yomu.fyi/post/5-career-myths-from-women-in-engineering.md)
- Company: [Two Sigma](https://yomu.fyi/company/two-sigma.md)
- Author: Emily Majewski
- Published: Mar 17, 2026
- Type: Explainer

Three engineering leaders at Two Sigma—Angela Wang, Cecilia Ye, and Mae Santos—challenge conventional career progression models by examining five persistent industry myths. Diverse transitions across civil engineering, enterprise architecture, and customer-facing solutions architecture supplied foundational skills for modern roles in feature engineering, reliability engineering, and real-time trading optimizers. The leaders emphasize that deep domain expertise stems from direct proximity to an active system rather than cumulative years of tenure. Beyond core programming capabilities, proactive contributions such as upgrading operational standards and cultivating peer relationships drive career opportunities and leadership recognition. Adapting to emerging technologies and maintaining consistent execution allows engineers to build credibility across evolving technical environments.


[Newer posts](https://yomu.fyi/topic/reliability/page/6.md) · [Older posts](https://yomu.fyi/topic/reliability/page/8.md)