Migrating Etsy’s database sharding to Vitess

Summary

Etsy migrated its long-running MySQL shard management to Vitess vindexes after an architecture spanning about 1,000 tables and shards, 425 TB of data, and roughly 1.7 million requests per second. The legacy ORM used a single index database to map records to shards, creating a scaling bottleneck and single point of failure. To avoid moving existing data, engineers built custom SQLite lookup and hybrid vindexes, changed new shard assignments to match Vitess’s hash algorithm, and introduced routing incrementally table by table with traffic ramps, rollback, and query compatibility testing. After five years, about 2,500 pull requests, and 6,000 queries, the migration removed the index database dependency, made scaling operations take days rather than months, simplified developer access, and kept database performance largely unchanged without a mass data migration.

Takeaways

1. 1

   Custom SQLite and hybrid vindexes preserved Etsy’s random legacy shard mappings while allowing new IDs to use the Vitess hash algorithm.

2. 2

   Etsy disabled scatter queries by default because an unintentionally broad query could be sent to all 1,000 shards; approved scatter queries nevertheless reduced one model’s query time from about 2 seconds to 20 milliseconds.

3. 3

   The completed migration removed the index database as a single point of failure, reduced scaling operations from months to days, and avoided a massive data migration while keeping performance largely unchanged.