10 Tips for Building Resilient Payment Systems

Summary

The post distills lessons from five years working on Shopify’s payment infrastructure into guidance for building systems that tolerate failure and large-scale traffic. It frames the advice partly around onboarding developers who were new to payments or Shopify’s scale, and notes that load testing for BFCM 2021 began in July with globally distributed load generation. It recommends low timeouts across HTTP clients and data stores, circuit breakers such as Semian, and capacity reasoning with Little’s Law, rate limiting, and load shedding. Monitoring should cover latency, traffic, errors, and saturation, while structured, machine-readable logs make centralized records searchable; load tests use benchmark gateways that mimic production partner latency and capacity. For incidents, Shopify uses defined roles, Slack-based coordination, and retrospectives that turn incorrect assumptions into safeguards and training material.

Context

The guidance addresses the difficulty of building resilient payment systems at Shopify’s scale, where new team members may lack experience with payments or large-scale systems. It also acknowledges that failure cannot be completely avoided and that merchants depend on Shopify Payments for accepting payments.

Approach / What changed

The post presents operational techniques including low timeouts, circuit breakers, capacity analysis, rate limiting, load shedding, monitoring, structured logging, production-like payment load testing, defined incident roles, and post-incident retrospectives.

Takeaways

• Ruby’s Net::HTTP defaults to 60 seconds for connection, write, and read timeouts; the post suggests one second for opening a connection and five seconds for writes, reads, or queries as a starting point.
• Shopify adds the merchant’s country code to a payment endpoint’s host-and-port circuit-breaker identifier so a local outage in one country does not affect transactions for merchants elsewhere.
• Shopify aims to hold incident retrospectives within a week, examining what happened, which assumptions were wrong, and which safeguards or learning materials should follow.

Related reading