Improving storage efficiency in Magic Pocket, our immutable blob store

Summary

After introducing a direct erasure-coding service, Dropbox encountered severe storage fragmentation in its immutable blob store, Magic Pocket, due to a large number of under-filled volumes. Because their legacy compaction strategy was only designed to top off near-full volumes, the team created new multi-strategy compaction methods to aggressively consolidate sparse volumes. These approaches leverage dynamic programming and streaming pipelines to rapidly reclaim unused disk capacity at exabyte scale.

Takeaways

1. 1

   In immutable blob stores, deleted data is not freed immediately and requires active compaction to rewrite live blobs into new volumes and retire fragmented ones.

2. 2

   Traditional host-and-donor compaction (L1) works well in steady state but struggles when a system accumulates a long tail of severely under-filled volumes.

3. 3

   Dropbox introduced L2 (a dynamic programming packing algorithm) and L3 (a continuous streaming re-encoding pipeline) to consolidate sparse volumes two to three times faster than before.