Loading…
Successfully Merging the Work of 1000+ Developers
2023-10-18
- Source
- Shopify
- Published
- Added to Yomu
Summary
Shopify’s single monolith supports changes 40 times a day and roughly 400 commits to master daily, making safe trunk-based development difficult at the scale of 1,000+ developers. Its first Merge Queue limited deployment drift and batch size, but queued pull requests did not receive CI, soft conflicts could break master, and a browser extension enabled accidental direct merges. Merge Queue v2 replaces that workflow with GitHub comments and webhooks, programmatic branch protection, an emergency /shipit --emergency path, and a predictive branch where queued changes receive CI before merging. A reconciliation algorithm synchronizes that branch through GitHub’s GraphQL API, while failure-tolerance thresholds handle flaky tests and remove repeatedly failing pull requests. CI starts when requests enter the queue; batches of eight and three concurrently tested batches balance throughput, risk, and resource use, while the changes improved user experience, deployment safety, and deploy throughput.
Context
Shopify needed to safely merge and deploy hundreds of daily commits across a large team and single codebase. Merge Queue v1 did not run CI while pull requests waited, allowing soft conflicts and broken merges; incidents could also create queues of more than 50 pull requests, while the browser extension enabled accidental direct merges.
Approach / What changed
Merge Queue v2 integrates the merge workflow with GitHub through comments, webhooks, GraphQL, and programmatic branch protection. It uses a predictive branch for pre-merge CI, reconciliation to maintain the desired GitHub state, failure-tolerance thresholds for flaky tests, an emergency bypass command, and bounded batches to improve deployment throughput.
Takeaways
- Merge Queue v2 starts CI as soon as a pull request enters the queue and uses a predictive branch to test the prospective state of master before merging.
- With an assumed 25% test flakiness rate, a failure tolerance of 3 reduces the false-positive probability to 0.39%; the first failing pull request is removed after a fourth consecutive failure.
- Shopify chose batches of eight pull requests and runs CI on three batches at a time, balancing deployment throughput and risk while limiting CI resource use.