Future Proofing Our Cloud Storage Usage

Summary

Shopify’s growth from 375,000 to over 600,000 merchants coincided with about 450,000 S3 operations per second and rising error rates plus read and write latency. Merchant-uploaded images and theme installations exposed a partitioning problem: S3 groups keys by shortest common asset prefixes, and abrupt throughput spikes could trigger SlowDown failures across a range of shops. Black-box observations suggested affected keyspaces were unlikely to recur, while newer shops were more often involved, though the latter was explicitly described as a shaky inference. Shopify prepended a hash-generated digest to new shops’ asset keys, distributing writes across more partitions instead of concentrating them under shop-specific prefixes. After the first post-change SlowDown event, exceptions nearly disappeared, while median latency for digested assets fell about 60% and the 95th percentile about 25%.

Takeaways

1. 1

   S3 partitions data using shortest common asset prefixes, and an abrupt throughput-limit breach can cause all operations on a partition to return AWS::S3::Errors::SlowDown.

2. 2

   Adding hash-generated entropy before new shops’ asset prefixes distributed writes across multiple partitions and reduced the chance that one partition would constrain a shop’s operations.

3. 3

   After the change, SlowDown exceptions almost disappeared; median latency for digested assets fell about 60%, while 95th-percentile latency fell about 25%.

FAQ