Planning in Bets: Risk Mitigation at Scale

Summary

Preparing for BFCM and other high-traffic events, Shopify describes risk mitigation for a globally distributed, complex, interconnected infrastructure platform that peaked at 75.98M requests per minute, or 1.27M per second. At this scale, identifying and mitigating every possible failure is impossible, so the process starts with “what could go wrong” exercises, then uses voting, expert review of likelihood and severity, and assigned ownership to prioritize action. Decision makers are empowered to weigh risks and rewards, while summarized findings, including key risks and single points of failure, are shared to maintain alignment and awareness. The post frames engineering choices as probabilistic bets: decision quality should be judged separately from outcomes, and uncertainty cannot be eliminated, only addressed through thoughtful prioritization and communication.

Context

Shopify prepares for BFCM and other high-traffic events across a large-scale infrastructure platform that is complex, interconnected, and globally distributed. At the stated scale, no single person can oversee the full design and detail, and it is impossible to identify and mitigate every possible risk with finite time and resources.

Approach / What changed

The process uses four questions: identify risks through “what could go wrong” exercises; prioritize them through voting and expert discussion of likelihood and severity; assign decision-making authority and action ownership; and communicate key findings, risks, and single points of failure to stakeholders. Probability is used to distinguish decision quality from eventual outcomes.

Takeaways

• “What could go wrong” exercises invite people across the platform to identify technology, operational, and other risks, creating broad visibility into possible problems.
• Risk prioritization combines voting with technical-expert review of likelihood and severity, followed by decisions about mitigation and ownership of each action item.
• The process treats engineering decisions as probabilistic bets: a good decision can have an unlucky outcome, and a bad decision can have a lucky one.

Related reading