Copy-and-Replay vs Branch-at-a-Point: Two Backup Recovery Paths
A side-by-side comparison of copy-and-replay recovery and branch-at-a-point restores, showing why restore time scales with data size in one model and not the other.
Loading…
Topic
Reliability
287 posts about Reliability. Every summary links to the original.
- 287 posts
- 45 companies
- 2 explainers
See the idea
Reliability explainers
The SQS Decision Most Diagrams Skip: Normal Work vs. Broken Work
Amazon SQS connects producers and workers, but its real design test is failure. Compare the processing path with the signals and controls needed when queue operations break.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Building rock-solid Stripe integrations: A developer's guide to success
The guide presents ten practices for building production-ready Stripe integrations, covering payment forms, subscriptions, and operational concerns. It recommends verifying webhook signatures with the Stripe-Signature header and endpoint secret, recording event IDs for idempotency, and returning a fast 2xx response while processing asynchronously. Test and live API keys, webhook endpoints, logs, and monitoring should remain separate, while error handling should distinguish card, request, and API failures and use exponential backoff for retries. The guide also recommends testing successful, declined, insufficient-funds, and 3DS scenarios with Stripe test cards, alongside simulated webhook events. Its conclusion emphasizes edge-case planning, logging, monitoring, and ongoing review as foundations for reliable, secure payment processing that can scale with business needs.
Building resilient webhook handlers in AWS: Implementing DLQs for Stripe events
Reliable Stripe webhook processing must account for lost deliveries from network or service outages, out-of-order events, and duplicates caused by Stripe retries. The proposed AWS architecture uses API Gateway for signature validation and throttling, an SQS FIFO queue for ordered delivery and content-based deduplication, Lambda for processing, DynamoDB for event-ID idempotency, and an SQS DLQ for failed messages. In the CloudFormation example, the main queue has a 300-second visibility timeout and sends messages to the FIFO DLQ after three receives; DynamoDB records expire through a seven-day TTL. Lambda retries failures with exponential backoff, while CloudWatch monitors queue depth, latency, and errors. The design is presented as a scalable foundation, with multi-region failover available at added cost and complexity, though single-region deployment may suffice for many applications.
New to Stripe? Learn the key concepts for software developers.
The guide introduces software developers to Stripe’s main objects and workflows for integrating payments, recurring billing, and related account activity. It begins with Payment Intents, which specify an amount and currency, track a payment’s lifecycle, and produce a client secret for completing the frontend flow, then covers Payment Methods and Customers, including Setup Intents and attaching a default payment method. For subscriptions, it distinguishes Products, Prices, and Subscriptions, and explains using events and webhooks to react to asynchronous payment and subscription changes while verifying webhook signatures. It also describes Disputes, refunds, and expanded API responses, including retrieving related resources in one call. The guide recommends starting in test mode and using dashboard logs while consulting Stripe’s documentation and changelog as the platform evolves.
Canva incident report: API Gateway outage
On November 12, 2024, canva.com suffered an outage lasting approximately one hour due to cascading API Gateway failures. The disruption began when a stale Cloudflare routing rule induced severe latency and packet loss between Ashburn and Singapore during an editor asset deployment. Cloudflare's concurrent streaming mechanism consolidated over 270,000 user requests for a single delayed JavaScript chunk; when the asset finally loaded, clients simultaneously generated a 1.5 million requests-per-second thundering herd to Canva's API Gateway. This surge collided with a telemetry performance regression involving lock contention on Netty event loop threads, exhausting off-heap memory and causing the Linux Out Of Memory Killer to terminate API Gateway containers. Canva mitigated the failure by temporarily blocking all traffic at the CDN level, stabilizing replacement Amazon ECS tasks, and gradually restoring traffic under strict rate limits.
Crush errors with Sandbox testing
Payment integrations can encounter errors caused by external events, such as network issues or outages, or by bugs in application code, and unhandled failures can stall checkout. The post presents Stripe Sandboxes as isolated environments for reproducing live-integration failures without changing live payment settings or traffic, optionally copying settings when a sandbox is created. Developers can use Workbench’s Errors and Logs views to diagnose a card_declined response, then route test requests through sandbox keys and a test card. On the Python backend, a try/except implementation catches Stripe errors and generic exceptions and returns an error response instead of leaving the payment page stalled. Sandboxes remain separate after creation, but each Stripe account can have at most five, so unused environments may be deleted or have their test data cleared while settings are preserved.
Testing Connect onboarding with Sandboxes
Stripe Connect testing can become unwieldy as platforms cover payment methods, US states, international countries, and merchant verification conditions, while the existing test mode stays synchronized with live settings. Sandboxes provide an isolated environment that can replicate a platform’s live configuration without affecting live payment traffic. The workflow covers creating a Sandbox, confirming that Connect settings such as negative-liability handling were preserved, and onboarding test connected accounts through Stripe-provided or custom flows. To reproduce a website-verification failure, a custom account can use “https://inaccessible.stripe.com”; after testing an update to “https://accessible.stripe.com” in the Sandbox, the platform can apply the fix to its production onboarding flow.
How do I store inventory data in my Stripe application
The DevRel Swag Store uses Stripe payments and AWS services to keep product inventory accurate and visible in near real time. Stripe products retain core payment information, while Amazon DynamoDB stores inventory and other attributes, keyed by Stripe product ID and store ID; EventBridge routes payment events to Lambda, which atomically decrements stock and publishes updates through AWS IoT Core to the frontend. DynamoDB condition expressions prevent decrements when stock is insufficient, while post-payment validation and refunds address the delay before a Stripe Payment Link is disabled, although Stripe fees may be non-refundable. A custom Payment Intent flow can check stock before payment, authorize funds with manual capture for up to seven days, update inventory after payment success, and then capture the charge.
Japan community highlights: Effective testing and security
This article reports practical lessons from two September 2024 JP_Stripes events in Aizuwakamatsu and Sapporo for developing and operating Stripe-integrated services more efficiently and at lower cost. Sandboxes let teams create up to five separate test environments, reproduce payment failures and state transitions through Stripe's API, and use CI-specific workspaces without production access. Stripe Connect examples show how a three-person codoc team launched in nine months by staggering account creation and embedding payout and payment-management interfaces. For fraud prevention, the article recommends Radar or Radar for Teams with webhook automation, including early-fraud-warning events and preemptive cancellation or refunds when dispute fees exceed transaction value. Together, these cases emphasize using managed Stripe capabilities and community-shared implementation experience to reduce testing effort, UI work, and fraud-related costs.
Turbocharging GrabUnlimited with Temporal
Grab’s GrabUnlimited flagship membership program struggled as subscribers grew by over 1000% from January 2022 to June 2023. Its original design used a daily cron job for renewals, Amazon’s SQS to distribute state handlers, and five-minute Redis locks, but outages, corrupted states, concurrent cancellation and renewal, missed benefits, and non-idempotent retries created operational problems. The team migrated the service to a Temporal workflow-based system, enhancing its ability to handle millions of subscribers with greater efficiency and resilience. The source reports an 80% reduction in open production incidents and says the change improved the membership experience for users. This first part focuses on designing a robust and scalable membership system, while migration challenges and lessons are reserved for a subsequent installment.
Adding payments to your LLM agentic workflows
Agentic workflows combine large language models with function calling to turn prompts into sequences of actions, but connecting those actions to financial services requires controlled API access. The Stripe agent toolkit integrates Stripe with Vercel’s AI SDK, LangChain, and CrewAI, supports function-calling LLM providers, and is built on Stripe’s Node.js and Python SDKs. It exposes configurable tools for tasks such as invoicing, creating and sharing payment links, and using Issuing virtual cards with spending limits, authorization checks, and deactivation after purchase. Middleware can record input and completion token counts as billing events for usage-based customer billing. Because agent behavior is non-deterministic, the post recommends test mode, evaluations, restricted keys, limited tool sets, and reduced request and response shapes to improve focus and constrain failures.
Testing subscriptions with Stripe Test Clocks and Workbench
Stripe subscriptions require testing scenarios that unfold across billing cycles, and the post presents Test Clocks and Workbench as no-additional-cost test-mode tools for doing so without waiting real time. It provisions three monthly recurring product tiers, creates a test clock with a Unix Epoch FrozenTime, and attaches newly created customers and subscriptions to that clock using the Stripe .NET SDK. Advancing the clock by one month updates time-dependent objects, allowing Workbench Inspector logs and events such as invoice.created and customer.subscription.updated to be examined alongside payment and subscription state. A second simulation attaches pm_card_chargeCustomerFail, makes it the default payment method, and advances to the next billing period. The resulting failed invoice, canceled subscription, and customer.subscription.deleted cancellation_details show how these tools reveal the events and properties an integration should test.
Avoiding test mode tangles with Stripe Sandboxes
Stripe integrations can become difficult to test when an account has only one test mode, because unrelated work can interfere and sensitive configuration must remain protected. The post presents Stripe Sandboxes as separate testing areas and shows a Node.js Express application authenticating with a sandbox secret key stored in a .env environment variable. It creates payment intents through /create-payment-intent and retrieves account details through /account, then uses Jest and supertest to verify the sandbox identity, including the expected display name dev-sandbox. A GitHub Actions workflow injects STRIPE_API_KEY from repository secrets, installs dependencies, and runs tests on pushes to the main branch, while .gitignore excludes environment files. The conclusion is that isolated sandboxes and automated checks reduce test tangles and help prevent calls to the wrong environment.
Advanced error handling patterns for Stripe enterprise developers
The post presents advanced error-handling patterns for Stripe integrations that face high request volumes, temporary API slowdowns, connectivity problems, and webhook delivery failures. It uses Stripe Workbench to inspect account-level API activity, traffic spikes, error rates, API versions, and webhook activity; one example links a 429 response to a scheduled job that created usage records across thousands of accounts simultaneously. The implementation combines request spacing with Limiter, a p-queue concurrency cap of five, and a token bucket configured for 100 tokens per 60 seconds. Stripe’s stated limits are 100 read and 100 write operations per second in live mode and 25 of each in test mode, with stricter limits for some resources. The reported before-and-after results show the combined controls processing high API-call volume within those limits, reducing errors and producing more predictable interaction.
Simple error handling strategies with Stripe Workbench
Stripe Workbench addresses the difficulty of finding and understanding errors produced by Stripe API invocations. It gives developers a view of recent errors, groups repeated failures, shows their frequency, and links summaries to request logs and actionable resolution guidance. The article describes Stripe error categories such as StripeCardError, StripeInvalidRequestError, and StripeConnectionError, then demonstrates Node.js try/catch handling that branches on the error type. It also covers webhook responses for payment_intent.payment_failed events, including retrieving event.data.object.last_payment_error, and explains how objects such as PaymentIntents, Setup Intents, Invoices, Setup Attempts, Payouts, and Refunds retain failure information for later inspection. Combined, these mechanisms support targeted responses, troubleshooting through detailed request and response data, and more reliable Stripe integrations.
Bringing your Stripe objects to life with Workbench
The post demonstrates how Stripe Workbench’s Inspector can be used to examine a PaymentIntent throughout its lifecycle and diagnose payment workflow issues. It maps states including requires_payment_method, processing, requires_action, succeeded, and canceled, showing how the status property changes as inputs and decisions move the object through a state machine. The walkthrough creates and confirms test PaymentIntents with Stripe Shell commands, including a 3DS-required card that produces a requires_action state and a use_stripe_sdk next_action. It also describes reacting to state changes through webhook endpoints or Amazon EventBridge, using the Inspector to identify friction such as PaymentIntents stalled at requires_action and to debug abandoned payments.
Office Hours with Engineering Managing Director Mae Santos
Mae Santos, Head of Critical Applications and Data Reliability Engineering at Two Sigma, leads a global team across New York, Houston, London, and Tokyo to support critical applications and data pipelines. While software teams frequently prioritize functional features over operational stability, designing systems with reliability in mind from the beginning allows for progressive enhancements without complete rewrites. System architecture should be guided by concrete goals across four key dimensions: availability, observability, scalability, and supportability. In addition, mapping out interconnections among systems, infrastructure, data, and human processes helps teams manage dependencies effectively. Applying automation across support and delivery workflows further reinforces reliability by minimizing human error and accelerating execution.
Optimizing Data Reliability: Heroku Connect & Drift Detection
Heroku Connect syncs Salesforce data with Heroku Postgres, but primary polling can miss changes when long-running Salesforce automations commit after the polling window advances, creating drift. The service uses secondary polls over the same bounds as primary polls to find and repair missed changes, with Bulk API or SOAP API selected according to dataset size and retries for failed polls. Recent improvements change Bulk API drift handling by creating one secondary bulk job for each primary bulk job, then adding corresponding batches instead of creating many jobs. Secondary tasks are now scheduled and executed with limited concurrency, reducing simultaneous database connections consumed while Postgres is updated and preserving capacity for applications. The post says these changes reduce load on Salesforce and improve operational reliability during heavy synchronization, while noting that Heroku Connect remains asynchronous rather than truly real-time.
No version left behind: Our epic journey of GitLab upgrades
Grab's self-hosted GitLab instance fell approximately 14 months behind official releases, serving thousands of engineers and monorepos reaching up to 39TB in total footprint. To address security updates and system stability, the team established a structured upgrade routine using Terraform, Packer, and Ansible across GitLab's 5,000-user reference architecture. They implemented staged deployments, upgraded stateful Gitaly nodes via in-place rotation, and resolved primary node clustering bottlenecks under Praefect. By utilizing official GitLab Upgrade Paths and parallelizing component deployments, the team caught up on 24 months of releases in 11 months and reduced release lag from 396 days down to 35 days.
Ensuring data reliability and observability in risk systems
Grab's in-house risk management platform, GrabDefence, relies on ingesting large volumes of upstream data to drive real-time heuristic rules and data science models. To prevent data discrepancies and missing information from degrading fraud detection, the team implemented a real-time data observability pipeline. The solution uses Apache Flink SQL alongside custom table functions like JSONEXPLOAD to flatten nested JSON structures and aggregate metrics over 5-minute tumbling windows. Processed metrics are sent to Datadog, where counters are grouped by source stream and evaluated using anomaly detection algorithms. Alerts are routed directly to Slack, reducing anomaly detection latency from days or weeks down to within the same day or hour.
Kafka on Kubernetes: Reloaded for fault tolerance
Grab's real-time data streaming platform, Coban, operates Kafka on AWS Elastic Kubernetes Service using Strimzi, allocating an entire EC2 worker node with NVMe instance store volumes to each broker. An initial architecture suffered from client connection errors, broken Network Load Balancer target groups, and zombie Persistent Volume Claims when worker nodes terminated. To achieve automated fault tolerance, the team integrated the AWS Node Termination Handler in Queue Processor mode with Auto Scaling lifecycle hooks, ensuring Kafka receives a SIGTERM to migrate partition leadership gracefully before shutdown. They also introduced the Kubernetes Cluster Autoscaler to dynamically provision replacement nodes during maintenance events and used the AWS Load Balancer Controller with TargetGroupBinding custom resources to dynamically update load balancer targets using IP mode.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.