Loading…
Performance Testing At Scale—for BFCM and Beyond
2023-10-18
- Source
- Shopify
- Published
- Added to Yomu
Summary
Shopify treats Black Friday Cyber Monday (BFCM) as its most critical period, and uses full-scale BFCM Scale Testing to assess whether the platform can handle peak traffic. Preparation combines capacity planning, resiliency testing, single-application performance testing, and repeated exercises that scale core services to their BFCM profiles before generating load. The hybrid architecture centers on Shopify Core, a monolithic Ruby application supported by services for storefront rendering, analytics, fraud analysis, payments, webhooks, and more. Five load flows cover browsing and buying, admin activity, flash sales, Storefront APIs, and Hydrogen and Oxygen, with Cronograma layering requests gradually and providing an abort switch. The exercises surfaced connection-count pressures, a Kubernetes ingress-nginx endpoint limit, and uneven load-balancer distribution; mitigations included refactoring, scaling dependent services, increasing pod capacity, and dedicated nginx node pools.
Context
BFCM is Shopify’s most critical time of year, bringing unprecedented milestones and the highest levels of scale for Shopify and its merchants. The testing is intended to ensure the ecommerce platform can handle BFCM traffic and to identify issues before the event.
Approach / What changed
The process scales the core platform and selected subsystems to BFCM profiles, runs staged load tests, scales back to baseline, analyzes findings, and repeats the cycle throughout the year. Tests use five traffic flows executed by Cronograma, with gradually increasing and layered load, continuous platform monitoring, and an abort switch. Teams also address discovered bottlenecks through service scaling, architectural refactoring, increased pod capacity, and dedicated nginx node pools.
Takeaways
- Cloud capacity is part of the preparation: Shopify worked with its cloud vendor to secure the virtual machines needed to expand an already-large footprint by 200 percent.
- Cronograma models browsing and buying, admin, flash sale, Storefront API, and Hydrogen and Oxygen journeys; an individual flow can make 25 to 50 HTTP calls.
- A 1000-endpoint ingress-nginx limit prevented all desired Shopify Core pods from receiving traffic. Increasing each pod’s CPU, memory, and web-worker capacity kept pods per cluster below that limit while preserving capacity.