---
title: "Future Proofing Our Cloud Storage Usage"
description: "Shopify’s growth from 375,000 to over 600,000 merchants coincided with about 450,000 S3 operations per second and rising error rates plus read and write latency. Merchant-uploaded images and theme installations exposed a partitioning problem: S3 groups keys by shortest common asset prefixes, and abrupt throughput spikes could trigger SlowDown failures across a range of shops. Black-box observations suggested affected keyspaces were unlikely to recur, while newer shops were more often involved, though the latter was explicitly described as a shaky inference. Shopify prepended a hash-generated digest to new shops’ asset keys, distributing writes across more partitions instead of concentrating them under shop-specific prefixes. After the first post-change SlowDown event, exceptions nearly disappeared, while median latency for digested assets fell about 60% and the 95th percentile about 25%."
---

# Future Proofing Our Cloud Storage Usage

[Shopify](https://yomu.fyi/company/shopify) · Andrew Louis · Feb 26, 2018

**Type:** Problem & solution

## Summary

Shopify’s growth from 375,000 to over 600,000 merchants coincided with about 450,000 S3 operations per second and rising error rates plus read and write latency. Merchant-uploaded images and theme installations exposed a partitioning problem: S3 groups keys by shortest common asset prefixes, and abrupt throughput spikes could trigger SlowDown failures across a range of shops. Black-box observations suggested affected keyspaces were unlikely to recur, while newer shops were more often involved, though the latter was explicitly described as a shaky inference. Shopify prepended a hash-generated digest to new shops’ asset keys, distributing writes across more partitions instead of concentrating them under shop-specific prefixes. After the first post-change SlowDown event, exceptions nearly disappeared, while median latency for digested assets fell about 60% and the 95th percentile about 25%.

## Context

The post addresses increasing S3 errors and latency in Shopify’s merchant flows, particularly product-image uploads and theme installations. It focuses on how S3 partitions asset keys and how abrupt request spikes can temporarily affect multiple shops.

## Approach / What changed

Shopify prepended a hash-generated digest to asset keys for new shops. For example, /s/2410/1317/files/theme.scss became /s/3582/2410/1317/files/theme.scss, distributing writes across multiple S3 partitions rather than concentrating them under the original prefix.

## Takeaways

- S3 partitions data using shortest common asset prefixes, and an abrupt throughput-limit breach can cause all operations on a partition to return AWS::S3::Errors::SlowDown.
- Adding hash-generated entropy before new shops’ asset prefixes distributed writes across multiple partitions and reduced the chance that one partition would constrain a shop’s operations.
- After the change, SlowDown exceptions almost disappeared; median latency for digested assets fell about 60%, while 95th-percentile latency fell about 25%.

## FAQ

### Why could activity from one shop affect a range of other shops?

Assets were stored under prefixes that could initially share a single S3 partition. If request rates abruptly exceeded that partition’s throughput limit, operations for the whole affected range could fail with SlowDown for minutes to a few hours.

### How did hash-based key prefixes improve reliability?

Shopify prepended a hash-generated digest to new shops’ asset keys, adding entropy and spreading writes across more partitions. A SlowDown event on one partition therefore did not significantly degrade a shop’s ability to write to the bucket.

**Tags:** [Cloud](https://yomu.fyi/topic/cloud), [Performance](https://yomu.fyi/topic/performance), [S3](https://yomu.fyi/topic/s3)

- Source: [Shopify](https://shopify.engineering/future-proofing-our-cloud-storage-usage)
- Source URL: https://shopify.engineering/future-proofing-our-cloud-storage-usage
- Ingested by Yomu: 2026-09-17T14:13:04.863Z

[Read original post](https://shopify.engineering/future-proofing-our-cloud-storage-usage)