---
title: "Reliability"
description: "289 posts about Reliability, summarised, each linking to the original."
---

# Reliability
> 289 posts about Reliability, summarised, each linking to the original.

## Articles

### [We’re Hiring a Principal Backend Engineer to Shape the Future of Medium](https://yomu.fyi/post/we-re-hiring-a-principal-backend-engineer-to-shape-the-future-of-mediu.md)
- Company: [Medium](https://yomu.fyi/company/medium.md)
- Author: Michael Margolis
- Published: Oct 22, 2025
- Type: Announcement

Medium is hiring a Principal Backend Engineer, its most senior individual-contributor level, to lead the evolution of backend systems supporting reading, writing, and discovery for millions of users. The role combines hands-on engineering with organizational leadership: the hire will define backend strategy, set quality and architectural standards, review critical designs and code, mentor engineers, and partner with engineering, design, data, and executive leadership. Its technical focus includes modernizing a large living codebase, improving reliability and scalability, and balancing long-term platform evolution with building, debugging, and refining systems directly. Medium says the position should help teams move faster and ship with confidence while keeping the technology fast, stable, and built for longevity; it is fully remote in the US and seeks engineers experienced with systems used by millions, technical debt, fan-outs, and complex edge cases.


### [The 10 Skills I Was Missing as a MongoDB User](https://yomu.fyi/post/the-10-skills-i-was-missing-as-a-mongodb-user.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Joel Lord
- Published: Oct 2, 2025
- Type: Explainer

Transitioning from relational databases to MongoDB often leads developers to replicate SQL habits, causing severe performance bottlenecks, brittle schemas, and operational challenges. Effective document design requires balancing referencing and embedding based on query access patterns and update frequencies, rather than placing every entity into separate collections or excessively nesting large documents. Database performance depends on matching indexes to specific query shapes and field orders, which developers can systematically verify by inspecting execution plans using explain plans. Utilizing the built-in aggregation framework allows filtering, grouping, and calculations to run natively inside the database, eliminating slow, manual data transformations in application code. Establishing early monitoring for latency, replication lag, memory usage, and cluster failover resilience ensures stability before adopting advanced tools like Atlas Search and Vector Search.


### [Carrying Complexity, Delivering Agility](https://yomu.fyi/post/carrying-complexity-delivering-agility.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Akshat Vig, Ashish Kumar
- Published: Sep 25, 2025
- Type: Explainer

MongoDB centers its engineering architecture around resilience, intelligence, and simplicity to minimize developer cognitive and operational burdens when building distributed applications. Security is enforced through dedicated clusters in isolated virtual private networks alongside Queryable Encryption, which allows equality and range queries on ciphertext without decryption keys ever leaving the client. High availability is built on replica sets across independent availability zones and multi-cloud topologies, utilizing consensus mechanisms that commit writes only after majority acknowledgment in the active term. Operational friction in artificial intelligence workloads is addressed by integrating vector search directly into the core query engine, eliminating brittle extract-transform-load pipelines and separate vector databases.


### [From Niche NoSQL to Enterprise Powerhouse: The Story of MongoDB's Evolution](https://yomu.fyi/post/from-niche-nosql-to-enterprise-powerhouse-the-story-of-mongodb-s-evolu.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Ashish Kumar
- Published: Sep 25, 2025
- Type: Explainer

Early perceptions categorized MongoDB alongside eventually consistent NoSQL databases, creating doubts regarding its suitability for mission-critical enterprise workloads. Over fifteen years of development, the platform addressed enterprise requirements by introducing replica sets with Raft-style consensus, native horizontal sharding, tunable read and write concerns, and multi-document ACID transactions across sharded clusters. Security capabilities were expanded through Role-Based Access Control, Field-Level Encryption, and Queryable Encryption supporting prefix, suffix, and substring searches. Current initiatives focus on decoupling compute and storage scaling, developing AI-driven tooling within the Application Modernization Platform to translate legacy relational applications to microservices, and optimizing query execution for non-denormalized data models. MongoDB Atlas now incorporates Atlas Search, Atlas Vector Search, native Stream Processing, and embedding generation powered by Voyage AI models.


### [MongoDB Community Edition to Atlas: A Migration Masterclass With BharatPE](https://yomu.fyi/post/mongodb-community-edition-to-atlas-a-migration-masterclass-with-bharat.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Nick Bell
- Published: Sep 21, 2025
- Type: Problem & solution

BharatPE, an Indian fintech processing over ₹12,000 crore monthly across 450 cities, operated a 45-terabyte self-hosted MongoDB Community Edition deployment. Managing three sharded three-node clusters resulted in uneven data distribution, high maintenance costs, scaling friction, and disaster recovery gaps under strict regulatory requirements. To modernize infrastructure, the database operations team partnered with MongoDB to execute a structured five-stage migration to MongoDB Atlas. The initiative utilized mongosync for secure data transition alongside environment mirroring and automated integrity validation scripts. Transitioning to MongoDB Atlas yielded a 40 percent improvement in query response times and provided automated failover backed by a 99.995 percent uptime SLA.


### [How we built it: Real-time analytics for Stripe Billing](https://yomu.fyi/post/how-we-built-it-real-time-analytics-for-stripe-billing.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Reed Trevelyan
- Published: Sep 16, 2025
- Type: Problem & solution

Stripe built a real-time streaming analytics system for Billing after customers said they needed to detect emerging behavior and adapt pricing quickly, replacing a batch pipeline whose subscription updates lagged by 24 hours on average. The architecture uses Apache Flink to maintain compressed subscription history and incrementally process analytics events, while a custom Apache Spark job generates initial Flink state from billions of historical events and supports validation and exports. A new Apache Pinot v2 query engine performs windowed aggregations and joins without offline preaggregation; most updates are processed in under one minute, nearly all reach users within 15 minutes, and production query latency is under 300 milliseconds. Metric-definition changes trigger historical recalculation while new events are buffered and later replayed, preserving a consistent Dashboard view during the transition.


### [Migrating to React Native's New Architecture](https://yomu.fyi/post/migrating-to-react-native-s-new-architecture.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Sep 5, 2025
- Type: Explainer

Shopify successfully migrated two largest apps, Shopify Mobile and Shopify Point of Sale, to React Native’s New Architecture while continuing weekly releases for millions of merchants. The strategy prioritized minimal code changes, dual-architecture compatibility, and performance and stability parity, with TopHat generating both architecture builds for every pull request. The team upgraded React Native first, updated or removed incompatible dependencies, and changed only native modules that failed under the new architecture rather than migrating all 40-plus modules to TurboModules. Migration issues included state batching, UIManager-related blank screens, shadow-tree manipulation, view flattening, and main-thread legacy module initialization; the reported outcome was maintained development velocity with no feature-development disruption, followed by planned optimization work.


### [Building production-ready agentic systems: Lessons from Shopify Sidekick](https://yomu.fyi/post/building-production-ready-agentic-systems-lessons-from-shopify-sidekic.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 26, 2025
- Type: Explainer

Shopify’s Sidekick evolved from a simple tool-calling system into an agentic assistant that helps merchants manage stores through natural language, exposing scaling challenges in architecture, evaluation, and training. As its inventory grew beyond 50 tools, the system prompt accumulated conflicting special cases and became difficult to maintain; JIT instructions instead return relevant guidance with tool data. For evaluation, the team uses Ground Truth Sets based on production distributions, human labels from at least three product experts, statistical agreement measures, specialized LLM judges calibrated against humans, and merchant simulation. Judge correlation rose from Cohen’s Kappa 0.02 to 0.61 against a human baseline of 0.69, while syntax validation improved from about 93% to 99% after reward-hacking fixes. GRPO training exposed opt-out, tag, and schema hacking, and the post recommends modularity, procedural plus semantic validation, user simulation, and iterative judge refinement for production agentic systems.


### [From naive webhooks to durable sync: Queue-based and event-driven data reconciliation patterns](https://yomu.fyi/post/from-naive-webhooks-to-durable-sync-queue-based-and-event-driven-data.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: James Beswick
- Published: Aug 22, 2025
- Type: Explainer

Maintaining consistent product data between an internal catalog and Stripe becomes difficult when outages, throttling, high volumes, and changes from both systems create reconciliation failures. The post compares a naive one-way flow, in which a Lambda calls the Stripe API with no persistent synchronization record, with a queue-based design that durably stores changes, controls processing rate, retries failures with exponential backoff, and sends persistent failures to dead-letter queues. For bidirectional synchronization, it contrasts periodic ETL, using extraction, transformation, and loading, with an event-driven model based on Stripe Event Destinations, EventBridge, SQS, and symmetrical events from internal systems. The conclusion is that durable, observable queues and event-driven processing provide buffering, auditability, near-real-time propagation, independent change processing, and more graceful scaling for distributed reconciliation.


### [Keeping product data aligned with Stripe as your systems scale](https://yomu.fyi/post/keeping-product-data-aligned-with-stripe-as-your-systems-scale.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: James Beswick
- Published: Aug 21, 2025
- Type: Explainer

Keeping internal product data aligned with Stripe becomes an architectural challenge as catalogs grow from hundreds to millions of items. The problem spans independently operated systems with different data models, API constraints, rate limits, availability guarantees, and consistency models, while products may change faster than periodic reconciliation can complete. Stripe's payment-focused model covers identifiers, descriptions, prices, currencies, billing terms, and subscription behavior, whereas internal systems may contain categories, inventory, rich content, technical specifications, media, and sensitive cost data. The series examines manual verification, simple scripts, and periodic jobs for smaller catalogs, then points to streaming, change data capture, intelligent partitioning, and conflict resolution for high-scale reconciliation. It concludes that cross-system synchronization requires deliberate mapping and automated architecture because perfect synchronization is difficult to guarantee in eventually consistent distributed systems.


### [Guardrails for money movement: Integrating Stripe MCP with Portia AI](https://yomu.fyi/post/guardrails-for-money-movement-integrating-stripe-mcp-with-portia-ai.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Mounir Mouawad
- Published: Aug 13, 2025
- Type: Tutorial

Portia AI presents a customer-refund agent that combines its guardrails with Stripe’s official Model Context Protocol server. The agent reads a refund request, compares it with a policy file, identifies the relevant Stripe customer and payment intent, and evaluates whether the request should be approved. Portia’s planner produces a multi-agent plan, while execution agents run it and persist outputs in plan state; an execution hook pauses before the Stripe create-refund call. A clarification presents the request and the agent’s rationale to a human, who can approve it in the CLI; approval allows the refund to proceed, while rejection exits without creating one. The example also handles Gmail OAuth and can send the customer an email describing the outcome.


### [How we built it: Jurisdiction resolution for Stripe Tax](https://yomu.fyi/post/how-we-built-it-jurisdiction-resolution-for-stripe-tax.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Erich Rentz
- Published: Jul 10, 2025
- Type: Explainer

Stripe Tax’s jurisdiction resolution system (JRS) determines which US taxing jurisdictions apply to a transaction, a difficult task because more than 16,000 combinations of rates and rules depend on intricate, changing boundaries. Its offline geographic information system cleans and standardizes boundary data, overlays states, counties, cities, and districts, and generates time-aware Stripe places of taxation (SPOTs), while the online system matches an address to the relevant SPOT. To reduce point-in-polygon latency, JRS indexes nested bounding boxes in a balanced R-tree rebuilt with the Sort-Tile-Recursive algorithm, narrowing candidates before applying the final polygon calculation; disjoint SPOTs are split and smaller boxes prioritized. Most states achieve address matching in a few milliseconds, with 95th-percentile latency below 10 milliseconds except South Carolina, while historical SPOT data creates an ongoing memory challenge.


### [Stay within limits: API rate-limit-friendly pattern for Stripe webhooks](https://yomu.fyi/post/stay-within-limits-api-rate-limit-friendly-pattern-for-stripe-webhooks.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Phil Leggetter
- Published: Jul 3, 2025
- Type: Tutorial

Stripe webhook handlers that treat events as signals and fetch the latest resource can preserve correctness when payloads are stale, partial, duplicated, or out of order, but bursts of events can drive excessive API traffic. The post describes Stripe’s general limit as 100 read requests per second and notes that exceeding it produces 429 responses. Its solution places Hookdeck Event Gateway between Stripe and the application: Hookdeck authenticates and queues incoming webhooks, throttles delivery, and lets the handler retrieve the current Stripe resource at a controlled pace. The Express.js flow verifies the Hookdeck signature, checks the event and resource ID, fetches the invoice with the Stripe SDK, and supports queue monitoring, alerts, and retries for backpressure.


### [Implementing scalable metered billing with Stripe: How Edgee handles billions of events](https://yomu.fyi/post/implementing-scalable-metered-billing-with-stripe-how-edgee-handles-bi.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Alex Casalboni
- Published: Jun 24, 2025
- Type: Problem & solution

Edgee needed usage-based billing for a business combining flat-rate subscriptions with charges for proxy requests and delivered data events, while processing billions of monthly requests across more than 100 edge locations. Its architecture captures web data at the edge through a proxy and records raw metering data centrally in Google BigQuery. To control several terabytes of data, Edgee aggregates usage by tenant and sends hourly Stripe meter events through AWS Lambda, with graduated pricing and idempotency handled by Stripe. A Go configuration defines meters, prices, flat-rate plans, and pricing tiers, while retry logic and idempotent updates support resilience during disruptions. The resulting billing system remains stateless and low-latency while providing accurate monthly invoicing and a simple hosted customer interface.


### [Resilient Import Maps - Better Theme Development and Beyond](https://yomu.fyi/post/resilient-import-maps-better-theme-development-and-beyond.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: May 28, 2025
- Type: Problem & solution

Shopify found that JavaScript modules and import maps failed in themes and App Blocks when modules loaded before a map, or when multiple maps collided. These restrictions reflected the requirement that module resolution remain stable, but real pages assembled by themes, apps, Shopify, and extensions could not reliably guarantee ordering or a single map. Working with web-platform contributors, Shopify proposed a reconciliation mechanism with one global import map, a resolved module set, and merging rules that preserve existing resolutions and drop conflicting or unsafe additions. The change reached the HTML specification and shipped in Chromium and WebKit; Chrome supports it from version 133 and Safari from 18.4. Shopify also uses es-modules-shim support for browsers without native support, while Firefox support is hoped for.


### [Introducing Stripe Workflows: Tailoring Payments to Your Business Needs](https://yomu.fyi/post/introducing-stripe-workflows-tailoring-payments-to-your-business-needs.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Ben Smith
- Published: May 7, 2025
- Type: Announcement

Stripe introduces Stripe Workflows, a Dashboard-based system for orchestrating Stripe products and APIs through responsive, end-to-end automation. It addresses the complexity of coordinating multiple Stripe products by replacing custom event-handling and dependency-management code with visual workflows that support real-time triggers, branching logic, dynamic fields, and more than 600 event triggers. Workflows includes execution monitoring with event-payload and step-level tracing, 30-day default log retention, exponential-backoff retries, built-in idempotency, and recursion protection capped at a depth of five. The post describes applications including reducing chargeback risk, routing high-value transactions for compliance review, and creating custom-priced subscriptions from finalized quotes without additional infrastructure.


### [Introducing Stripe's new public preview release channel](https://yomu.fyi/post/introducing-stripe-s-new-public-preview-release-channel.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Paul Anderson
- Published: May 7, 2025
- Type: Announcement

Stripe announces its new public preview release channel, which provides early access to API features before their General Availability launches. Preview features are described as operationally stable and production-ready, but they may receive breaking changes more often than GA features and may have geographic, onboarding, or approval restrictions. Each preview uses a distinct API version ending in preview, such as 2025-04-30.preview, selected through the Stripe-Version header; preview versions also appear in the API reference and changelog. Beta SDKs will use these preview versions and include preview features, with releases differentiated from GA counterparts by suffixes such as 5.2.0-beta.1. The example shows that a feature released only in preview is rejected by a GA version, requiring an explicit version change before use; this separation lets teams integrate earlier and provide feedback while avoiding unintended adoption of features with known restrictions.


### [Because nobody likes being charged twice](https://yomu.fyi/post/because-nobody-likes-being-charged-twice.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Ben Smith
- Published: Apr 10, 2025
- Type: Problem & solution

Payment workflows can produce duplicate charges when a customer retries after a connection drop while the backend is still retrying the original request. The post frames network timeouts, crashes, database locks, downstream errors, and user interruptions as normal distributed-systems failures that can leave payment state uncertain. It recommends generating a unique idempotency key for one specific action, sending it with the Stripe request, and reusing it on every retry; Stripe recognizes the key for 24 hours and returns the original response. It also recommends placing payment jobs in a queue such as Amazon SQS, where workers retry failures and route repeatedly failing messages to a dead-letter queue. Together, queues preserve work during temporary outages and idempotency makes retries safe, while careful key scope and timing avoid stale or cross-operation behavior.


### [Using an AWS microservice architecture for subscription management](https://yomu.fyi/post/using-an-aws-microservice-architecture-for-subscription-management.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: James Beswick
- Published: Apr 8, 2025
- Type: Explainer

Enterprise subscription management must handle mid-cycle plan changes, usage-based billing adjustments, synchronized service updates, atomicity, failed operations, data consistency, and auditability beyond basic recurring billing. This sample architecture uses an event-driven AWS design in which subscription changes are published to EventBridge and routed to Step Functions workflows. EventBridge provides at-least-once delivery, content-based filtering, dead-letter queue support, and integration with Step Functions, while Lambda functions validate changes, calculate Stripe proration, update external subscriptions, and trigger service-specific actions. Step Functions supplies retries with exponential backoff for StripeTemporaryError, DynamoDB stores current and historical subscription data using transactional updates, and Stripe calls use an event identifier as an idempotency key. The pattern is presented as a flexible foundation for reliability, maintainability, compensation for partial changes, and auditability in systems handling financial transactions and service provisioning.


### [Scaling Nextdoor’s Datastores: Part 4](https://yomu.fyi/post/scaling-nextdoor-s-datastores-part-4.md)
- Company: [Nextdoor](https://yomu.fyi/company/nextdoor.md)
- Author: Ronak Shah
- Published: Mar 19, 2025
- Type: Problem & solution

Look-aside caching systems can become inconsistent when concurrent database updates execute cache writes out of order, allowing stale data to overwrite newer modifications. To prevent these racing writes, Nextdoor introduced a unique, monotonic db\_version column to Postgres tables using database triggers that initialize version numbers on insert and increment them on update. Application updates retrieve this new version inside a transaction block and attach it as a metadata header to serialized cache values. Redis then executes custom Lua scripts, specifically set\_if\_version and del\_if\_version, to perform atomic conditional updates that reject any incoming payload with a version lower than or equal to the stored version. This serializable check ensures that out-of-order writes are dropped and the cache remains strictly aligned with the latest database state.


[Newer posts](https://yomu.fyi/topic/reliability/page/8.md) · [Older posts](https://yomu.fyi/topic/reliability/page/10.md)