---
title: "Reliability"
description: "289 posts about Reliability, summarised, each linking to the original."
---

# Reliability
> 289 posts about Reliability, summarised, each linking to the original.

## Articles

### [Unordered mode for GCS ClickPipes is now available](https://yomu.fyi/post/unordered-mode-for-gcs-clickpipes-is-now-available.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Marta Paes
- Published: Mar 16, 2026
- Type: Announcement

ClickHouse Cloud has added unordered mode to its Google Cloud Storage (GCS) ClickPipes connector, enabling continuous data ingestion regardless of file naming or arrival sequence. Standard GCS ClickPipes rely on polling every 30 seconds and require files to arrive in strict lexicographical order, which causes out-of-order data and backfills to be ignored. Unordered mode replaces polling by consuming OBJECT\_FINALIZE notifications from Google Cloud Pub/Sub whenever objects land in a bucket. State tracking is managed through a metadata store, and the connector automatically reprocesses batches upon failure while maintaining exactly-once insertion guarantees. Setup requires configuring a Pub/Sub topic and bucket notifications via tools like gcloud, followed by configuring the ClickPipe via the console or Terraform provider.


### [Supabase Storage: major performance, security, and reliability updates](https://yomu.fyi/post/supabase-storage-major-performance-security-and-reliability-updates.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: fabrizio
- Published: Mar 5, 2026
- Type: Announcement

Supabase Storage has shipped updates live in all regions for projects using Storage, targeting performance at scale, security vulnerabilities, and reliability issues. As projects reached tens of millions of objects, concurrent uploads exposed write bottlenecks, certain SQL deletes could orphan files, and a crafted path could escape the file backend’s storage root. The object-listing rewrite removes the prefixes table, six triggers, and twelve helper functions, derives folder structure from the objects table with a hybrid skip-scan algorithm, and uses cursor-based pagination; deep pagination on 60 million rows is up to 14.8 times faster without a write penalty. Direct SQL deletes are blocked by default through a statement-level trigger, while a configurable 30-second timeout supports cancellation of disconnected clients via native Postgres protocol. Idempotent migrations, TUS lock-race prevention, orphan-scanner fixes, OpenTelemetry metrics, and adapter and API bug fixes are included, with no code changes required.


### [PlanetScale protects production reliability with Bugbot](https://yomu.fyi/post/planetscale-protects-production-reliability-with-bugbot.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Cursor
- Published: Mar 2, 2026
- Type: Problem & solution

PlanetScale describes code review as the bottleneck created by rapidly increasing agent-generated code, while human review capacity remained fixed. To preserve production reliability, the company adopted Bugbot as a dedicated agentic review layer alongside its existing process. Bugbot surfaces semantic and logical issues that static analyzers, linters, and straightforward frontier-model reviews may miss, including premature completion states, non-converging asynchronous controllers, and database restart edge cases. Across more than 2,000 pull requests reviewed monthly, roughly 80% of Bugbot's comments are addressed before merge. PlanetScale says this prevents potential production downtime, saves the equivalent of two full-time engineers of review effort, and lets engineers focus on complex infrastructure work without sacrificing quality.


### [Towards Model-based Verification of a Key-Value Storage Engine](https://yomu.fyi/post/towards-model-based-verification-of-a-key-value-storage-engine.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Will Schultz, Murat Demirbas
- Published: Feb 27, 2026
- Type: Problem & solution

MongoDB developers formalized the interface boundary between their distributed transactions protocol and the underlying WiredTiger key-value storage engine using a compositional TLA+ specification. This modular model established an abstract contract to verify that the storage engine's concrete timestamp-based operations matched the semantics expected by the distributed transaction layer. To check conformance, a modified version of the TLC model checker generates a complete graph of reachable states for finite parameters and computes path coverings. Each identified path is translated into an automated test case consisting of a sequence of storage engine API calls. For a finite model with two keys and two transactions, this workflow generated 87,143 test cases that executed against WiredTiger in approximately 40 minutes.


### [Building a mental model for Stripe payments](https://yomu.fyi/post/building-a-mental-model-for-stripe-payments.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Allison Farris
- Published: Feb 19, 2026
- Type: Tutorial

An online bookshop checkout must coordinate card networks, issuing banks, fraud checks, regulatory requirements, currencies, and varied payment methods while completing in roughly 2–3 seconds. Stripe abstracts these concerns through a unified API, with the PaymentIntent representing the payment lifecycle from amount and currency collection through tokenization, authorization, capture, and settlement. The Payment Element and client-side tokenization keep sensitive card details away from the shop’s servers, while Stripe may perform risk checks and additional 3D Secure authentication. Because payments can change state minutes, hours, or days later through refunds and disputes, webhook events should be treated as the source of truth for synchronizing order and downstream systems without polling.


### [IBM and UC Berkeley Diagnose Why Enterprise Agents Fail Using IT-Bench and MAST](https://yomu.fyi/post/ibm-and-uc-berkeley-diagnose-why-enterprise-agents-fail-using-it-bench.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Ayhan Sebin, Rohan Arora, Saurabh Jha
- Published: Feb 18, 2026
- Type: Benchmark

IBM Research and UC Berkeley analyze how agentic LLM systems fail in enterprise IT automation, including incident triage, logs and metrics queries, and Kubernetes actions across long-horizon tool loops. To move beyond benchmark success rates, the analysis applies MAST (Multi-Agent System Failure Taxonomy) to execution traces, converting failures into structured signatures across system design, inter-agent misalignment, and task verification. It covers 310 ITBench SRE traces from Gemini-3-Flash, Kimi-K2, and GPT-OSS-120B, whose mean recalls were 75.5%, 28.6%, and 12.4%, respectively. Gemini averaged 2.6 failure modes per failed trace, compared with 4.7 for Kimi and 5.3 for GPT-OSS-120B, while FM-3.3, Incorrect Verification, was identified as the strongest predictor across models; the analysis recommends external verification, explicit termination controls, and clarification branches.


### [Supabase incident on February 12, 2026](https://yomu.fyi/post/supabase-incident-on-february-12-2026.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: paul\_copplestone
- Published: Feb 13, 2026
- Type: Incident / postmortem

Supabase reports a 3-hour-and-42-minute outage on February 12–13, 2026, affecting every service in its us-east-2 region, including Postgres, Auth, Data APIs, Edge Functions, Storage, and Realtime. A deployment of an internal monitoring service enabled AWS VPC Block Public Access in block-bidirectional mode, blocking internet gateway traffic across production VPCs in the region. The incident response was delayed because alerts pointed toward shared services, the relevant CloudTrail event was inconspicuous, pre-production lacked us-east-2, and infrastructure specialists were not involved initially. Supabase restored connectivity by rolling back and removing the monitoring stack, then audited regions, added AWS Organizations policies and IaC safeguards, and planned account isolation, external connectivity probes, production-parity testing, and improved escalation.


### [OpenEnv in Practice: Evaluating Tool-Using Agents in Real-World Environments](https://yomu.fyi/post/openenv-in-practice-evaluating-tool-using-agents-in-real-world-environ.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Christian Washington, Ankit Jasuja, Santosh Sah, Lewis Tunstall, ben burtenshaw
- Published: Feb 12, 2026
- Type: Explainer

OpenEnv is an open-source framework from Meta and Hugging Face for evaluating AI agents against real systems rather than simulations, with stateful environments that connect to tools and APIs. Turing's production-grade Calendar Gym applies this model to calendar operations, combining access control lists, partial visibility, temporal reasoning, and multi-step workflows in isolated sessions. Agents often handled individual actions but became less reliable on longer or ambiguous tasks: success was close to 90% with explicit calendar identifiers and roughly 40% when requests used natural-language descriptions. More than half of failures involved malformed arguments or incorrect action ordering despite selecting the correct tool, while structured validation, permission, and datetime errors provided concrete recovery signals.


### [Engineering at scale, with purpose.](https://yomu.fyi/post/engineering-at-scale-with-purpose.md)
- Company: [Thumbtack](https://yomu.fyi/company/thumbtack.md)
- Author: Thumbtack People Team
- Published: Feb 5, 2026
- Type: Explainer

Senior software engineer Brett Shouse outlines site reliability engineering initiatives and infrastructure modernization efforts underway at Thumbtack. The engineering organization is currently leading an operational project to migrate multiple self-hosted observability services onto a single, unified SaaS platform. This architectural transition consolidates application logs, distributed traces, and system metrics into one accessible view tailored for engineers, customer support staff, and company executives. Moving away from legacy self-hosted monitoring systems reduces systems administration overhead, mitigates alert fatigue, and lowers direct operational infrastructure costs. Concurrently, the reliability team tackles technical debt accumulated from rapid organizational growth by establishing structured incident response processes and automating repetitive operational toil.


### [Edge AI Made Easy: MongoDB and ObjectBox Data Synchronization](https://yomu.fyi/post/edge-ai-made-easy-mongodb-and-objectbox-data-synchronization.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Puja Roy
- Published: Feb 3, 2026
- Type: Announcement

Modern edge AI deployments require immediate real-time decision-making, local privacy, and offline operation while retaining the analytical scale of central cloud infrastructure. To address these hybrid demands, MongoDB and ObjectBox partnered to release the ObjectBox MongoDB Sync Connector as part of ObjectBox 5.0. ObjectBox operates as a lightweight on-device database with local vector search, multi-language bindings, and high-throughput I/O suited for resource-constrained hardware. In environments such as industrial factory floors and restaurant point-of-sale terminals, devices capture and evaluate operational data onsite without depending on active network connections. The native connector subsequently synchronizes critical operational datasets to MongoDB Atlas for long-term storage, fleet-wide monitoring, and machine learning model retraining.


### [The safety of speed: How we ship code 180 times per day](https://yomu.fyi/post/the-safety-of-speed-how-we-ship-code-180-times-per-day.md)
- Company: [Intercom](https://yomu.fyi/company/intercom.md)
- Author: Danny Fallon,Ryan Sherlock
- Published: Jan 27, 2026
- Type: Explainer

Intercom describes how it averages 180 production ships per workday in January 2026 while targeting 99.8+% availability, with an average 12-minute merge-to-production cycle. The post focuses on its Ruby on Rails monolith, deployed to three data-hosting regions through independent pipelines, and frames small batches as a way to limit accumulated risk. An automated pipeline builds a deployable slug, runs parallel CI, verifies it in pre-production with boot, CI, and Datadog Synthetics gates, then performs a staggered rollout with process-level rolling restarts. Safety also depends on engineer ownership, feature flags, production testing on a single host, outcome-focused heartbeat metrics, and rapid automatic or manual rollbacks. Incident reviews then harden the system, supporting the conclusion that speed is used to keep changes small, feedback immediate, and recovery fast rather than to trade stability for velocity.


### [Configuring Stripe using Terraform and AI agents](https://yomu.fyi/post/configuring-stripe-using-terraform-and-ai-agents.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Michael Selander
- Published: Jan 27, 2026
- Type: Tutorial

The post addresses the difficulty of using AI agents to configure Stripe resources when one-off API calls leave no durable source of truth for state, changes, or environment differences. It recommends using agents to author Terraform configuration rather than operating Stripe directly, so products, prices, and webhook endpoints become explicit, reviewable code. The example defines a Standard Plan with $20 monthly and $200 annual prices, selected webhook events, and outputs for both price IDs, then applies it with terraform plan and terraform apply. Terraform provides declarative, repeatable configuration supported by Git history, pull-request review, and environment-specific workspaces. Separate sandbox and livemode workspaces, paired with matching STRIPE\_API\_KEY values, are presented as a way to reduce accidental cross-environment changes.


### [AssetOpsBench: Bridging the Gap Between AI Agent Benchmarks and Industrial Reality](https://yomu.fyi/post/assetopsbench-bridging-the-gap-between-ai-agent-benchmarks-and-industr.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Dhaval Patel, James Rayfield, Saumya Ahuja, Chathurangi Shyalika, Shuxin Lin, Zhou, Ayhan Sebin
- Published: Jan 21, 2026
- Type: Benchmark

AssetOpsBench is an evaluation framework designed to benchmark agentic AI systems within industrial Asset Lifecycle Management operations, such as chillers and air handling units. The benchmark incorporates 2.3 million sensor telemetry points, 4,200 work orders, 53 structured failure modes, and over 140 curated scenarios spanning four agent roles. Systems are assessed across six qualitative criteria, including task completion, retrieval accuracy, result verification, sequence correctness, clarity and justification, and hallucination rate. Evaluation traces are analyzed using a trajectory-level diagnostic pipeline called TrajFM, which extracts and clusters failure patterns. In community tests covering over 300 agents, no evaluated model family reached the 85-point readiness threshold.


### [AprielGuard: A Guardrail for Safety and Adversarial Robustness in Modern LLM Systems](https://yomu.fyi/post/aprielguard-a-guardrail-for-safety-and-adversarial-robustness-in-moder.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Jaykumar Kasundra
- Published: Dec 23, 2025
- Type: Announcement

Modern language model systems face security and safety threats spanning prompt injection, jailbreaks, and tool manipulation across long-context agentic workflows. To address these vulnerabilities, researchers introduced AprielGuard, an 8B parameter causal decoder-only transformer based on a downscaled Apriel-1.5 Thinker Base model. The model detects 16 safety risk categories alongside binary adversarial attacks across standalone prompts, multi-turn dialogues, and agentic workflows. Trained on synthetic datasets created with Mixtral-8x7B, NVIDIA NeMo Curator, and SyGra, AprielGuard operates in both explainable reasoning and fast non-reasoning modes. Evaluation shows high precision and recall on safety and adversarial benchmarks, though trade-offs exist between explainability latency and non-reasoning performance.


### [From Python3.8 to Python3.10: Our Journey Through a Memory Leak](https://yomu.fyi/post/from-python3-8-to-python3-10-our-journey-through-a-memory-leak.md)
- Company: [Lyft](https://yomu.fyi/company/lyft.md)
- Author: Jay Patel
- Published: Dec 15, 2025
- Type: Incident / postmortem

During an initiative to upgrade Python services from version 3.8 to 3.10, Lyft engineers encountered severe latency spikes and timeouts in one service within their test environment. Profiling DynamoDB queries revealed that gevent thread joins were taking up to 30 seconds while pod memory consumption climbed steadily. To isolate the issue, engineers used an internal tracemalloc-based profiler triggered via SIGUSR2 signals, temporarily disabling Gunicorn preload to prevent workers from terminating prematurely on signal receipt. Memory traces pointed to botocore and an incompatibility between weakref.finalize and gevent monkey patching in urllib3 version 1.26.16, which prevented connections from returning to the pool. Downgrading urllib3 to version 1.26.15 immediately resolved both the timeouts and the memory leak before a permanent fix arrived in gevent and urllib3 updates.


### [Versioning in Stripe Workflows: Ship workflow changes with confidence](https://yomu.fyi/post/versioning-in-stripe-workflows-ship-workflow-changes-with-confidence.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Tanya Boiteau
- Published: Dec 10, 2025
- Type: Announcement

Stripe Workflows previously required teams to edit live workflow definitions directly, leaving no draft state, historical definitions, straightforward rollback, or reliable way to inspect which configuration shaped an older run. Versioning addresses this with a dedicated draft for iteration, immutable numbered published versions, and version history that can include descriptions, authors, status, and run counts. Publishing creates a complete snapshot containing steps, branches, conditions, variable references, calls, and configuration; new runs use it immediately, while in-progress runs finish on the version they started with. Each run records its exact version and exposes the workflow definition, including taken and untaken branches, while a past version can seed a new draft for rollback; versioning is available in Stripe Workflows, where new workflows include a draft by default.


### [Building production-ready Stripe subscriptions using Kiro powers](https://yomu.fyi/post/building-production-ready-stripe-subscriptions-using-kiro-powers.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Kalyani Koppisetti
- Published: Dec 4, 2025
- Type: Tutorial

Kiro's Stripe power is presented as a way to build a production-ready SaaS subscription flow while avoiding the fragmented output common with generic coding assistants. Using a natural-language specification for three monthly tiers and a three-month one-time option, Kiro creates a React frontend, Node.js backend API, CDN-hostable static assets, and server-side Stripe Checkout Session creation. The integration includes Stripe-specific steering files, MCP servers, and hooks, while the webhook handler preserves the raw request body, verifies HMAC signatures with stripe.webhooks.constructEvent(), rejects invalid events with HTTP 400, and routes lifecycle events such as checkout completion and subscription cancellation. The post concludes that this provides a secure, scalable foundation while leaving teams to customize business logic, including access provisioning and deprovisioning, instead of building payment infrastructure from scratch.


### [Engineering the right opportunities for Thumbtack Pros.](https://yomu.fyi/post/engineering-the-right-opportunities-for-thumbtack-pros.md)
- Company: [Thumbtack](https://yomu.fyi/company/thumbtack.md)
- Author: Ashmannsyngle
- Published: Dec 3, 2025
- Type: Explainer

Ashmann Syngle, a backend software engineer on Thumbtack's Pricing team, focuses on systems that connect local service professionals with homeowners. To improve matching and revenue operations, the team recently deployed new pricing features, launched experiments, and partnered closely with Data Science and Monetization Experience groups. Current engineering efforts center on infrastructure enhancements, specifically upgrading alerting and monitoring across owned services to raise system reliability and operational efficiency. Because the team maintains complex monetization systems within a codebase that has evolved over many years, engineers conduct deep system analyses to evaluate edge cases before implementing foundational platform modifications. Thumbtack supports these initiatives through a virtual-first operational model complemented by regular in-person offsites.


### [Run for your money: Engineering a treadmill that prints store credit](https://yomu.fyi/post/run-for-your-money-engineering-a-treadmill-that-prints-store-credit.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 25, 2025
- Type: Problem & solution

For the 2025 NYC marathon weekend, Shopify and Endorphins Running built a gamified treadmill that awarded in-store credit for maintaining a self-selected pace. Participants had 30 seconds to reach their target, then earned credit for each second spent within a defined pace zone, with a personalized barcode receipt redeemable through a Shopify POS UI extension. The implementation used a curved, motorless TrueForm Trainer, FTMS messages, a Python relay, Socket.IO, a local Remix application, receipt printing, and a production-hosted POS integration. Interval timestamps were normalized because treadmill messages arrived irregularly, while a state machine coordinated warm-up, gameplay, operator controls, database updates, discounts, and printing. The resulting experience attracted new customers, with 63% of participants identified as new customers, and its patterns could support other sensor-driven in-store incentives.


### [How we prepare Shopify for BFCM](https://yomu.fyi/post/how-we-prepare-shopify-for-bfcm.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 20, 2025
- Type: Explainer

Shopify’s BFCM readiness program addresses the challenge of operating critical commerce infrastructure at traffic levels informed by 2024 records, while preparing APIs with no prior holiday-traffic history. Starting in March, teams combined capacity planning on Google Cloud, an infrastructure roadmap, risk assessments, chaos-engineering Game Days, and a Resiliency Matrix covering failure modes, recovery objectives, runbooks, and on-call coverage. Genghis load tests and Toxiproxy network faults ran across three GCP regions, while five April–October scale tests exercised browsing, checkout, analytics, webhooks, regional failover, and cascading failures; testing reached 146 million requests per minute and a p99 target of 200 million. The work exposed and addressed Kafka partition, memory, timeout, routing, queueing, and failover issues, while the operational plan adds monitoring, 24/7 incident coverage, merchant communications, live tuning, and a post-BFCM review.


[Newer posts](https://yomu.fyi/topic/reliability/page/7.md) · [Older posts](https://yomu.fyi/topic/reliability/page/9.md)