How to Reliably Scale Your Data Platform for High Volumes

Summary

Shopify’s data platform prepared for BFCM 2020, when merchant sales exceeded $5.1 billion and platform throughput rose by an average of 150 percent. The Data Platform Engineering team focused on throughput risk, which grows with volume and is greatest near ingestion and processing, while prioritizing work through four service tiers. For Tier 1 and Tier 2 systems, engineers defined each service’s primary objective, identified adjustable controls such as job frequency, overlap intervals, CPU and memory requests, BigTable node pools, and ProxySQL connection queues, then used controlled load tests to expose capacity and recovery risks. The resulting work included cluster scaling, load-balancing adjustments, query and job tuning, and improved indices, schemas, dashboards, playbooks, SLOs, incident handling, and alerts. The post reports that BFCM 2020 completed without merchant-impacting incidents; only a handful of minor, self-healing incidents occurred, while monitoring and occasional tuning preserved spare capacity.

Context

BFCM generates unprecedented data and query volume, with forecasts of nearly twice 2019 traffic. The platform must provide data quickly and reliably to merchants, partners, and internal teams, and throughput risk is highest for ingestion and processing systems.

Approach / What changed

The team used tiered service priorities, defined primary objectives and SLOs, identified service-specific and compute controls, ran controlled load tests, reviewed alarms and recovery procedures, then adjusted infrastructure and tuning before the event.

Takeaways

• Longboat’s freshness objective requires every extracted dataset to be no older than the default eight-hour SLO; Reportify instead measures reliability against query latency.
• Tripling Longboat’s overlap interval simulated roughly three times more data to ingest without changing job frequency, providing a controlled throughput test.
• BFCM 2020 ended without incidents affecting merchants, buyers, or internal teams; the handful of minor incidents were mainly self-healing.

Related reading