Upgrading MySQL at Shopify

Summary

In September 2021, Shopify completed its cutover from Percona Server 5.7.21 to 5.7.32 across a MySQL fleet spanning hundreds of horizontally sharded databases and thousands of virtual machines. The upgrade targeted replication improvements from transaction writesets, along with security and stability fixes, while avoiding the larger risks of a major-version change. The Database Platform team extended Mason into a declarative topology system and used Chef roles to provision upgraded VMs, restore backups, run mysql_upgrade when required, configure replication, and populate replacement topologies. Extensive rollback testing exposed a critical downgrade bug; after patching 5.7.21, Shopify proceeded in staggered batches beginning August 25 and completed the final shards on September 7, while documenting a repeatable process intended to make future upgrades faster and safer.

Context

Shopify’s growing replica load, globally distributed replication topologies of up to four hops, and high-traffic events such as BFCM increased the risk of replication lag and reduced data availability after writer failures. The fleet’s size also made manual provisioning and upgrades highly laborious, while the team wanted a documented process that could be completed in weeks rather than months.

Approach / What changed

The team selected the highest available minor release in the Percona Server 5.7 series, 5.7.32, and extended Mason from a reactive self-healing service into a declarative system for reconciling each shard’s current and desired topology. Chef provisioning restored 5.7.21 backups onto 5.7.32 VMs, conditionally ran mysql_upgrade, configured replication, and supported staged replacement and takeover. The upgrade was rolled out in batches after benchmarking and rollback testing.

Takeaways

• Transaction writesets were the main technical motivation because they could preserve replication parallelism across Shopify’s multi-tier replication hierarchy.
• Rollback testing uncovered a critical downgrade bug that would otherwise have remained unknown; patching the 5.7.21 build provided a safer fallback.
• Mason’s declarative topology automation reduced upgrade and scaling toil across a fleet with per-shard differences in replica count, regions, CPU, and memory.

Related reading