Loading post…
ShopifyAlejandro Rodriguez SalamancaProblem & solution
The Unreasonable Effectiveness of Test Retries: An Android Monorepo Case Study
Summary
An Android monorepo expansion and larger team increased the number of tests run on master, exposing how flaky tests and infrastructure failures could erode CI confidence. Under an independence assumption, tests passing 99.95% each produced a theoretical full-suite pass rate of only 35% across 20 pipelines, while retrying each failed test once raised it to 99.95%. Shopify therefore added retries for setup failures, distinguishing failures retriable within a job from fatal failures requiring the test environment to reload, and retried failed tests up to three times. The system also flagged tests that passed after retry as warnings, since retries can hide reliability problems and slow builds. In practice, Android app pipeline pass rate rose from 31% to almost 90% after retry mechanisms and Gradle caching, while iOS rose from 67% to 97%.
Context
The work addressed declining CI pass rates after Shopify moved its Android apps and libraries to a monorepo and increased the Android team. The source describes extensive automated testing and notes that screenshot tests were flakier than other tests.
Approach / What changed
Failures were classified into retriable failures, which could be retried within the same job, and fatal failures, which required reloading the test environment. Failed tests were retried up to three times, and tests passing after a retry were reported to developers as warnings. The infrastructure also applied retries to setup operations such as Docker, Bundler, and git fetch, and added caching to Gradle builds to reduce interactions with external services.
Takeaways
With 100 tests per pipeline, 20 pipelines, and a 99.95% individual test pass rate, the theoretical full-suite pass rate was 35%; retrying once increased the theoretical rate to 99.95%.
The CI system separated failures that could be retried inside the current job from fatal failures requiring the test environment to be rebuilt, trading additional time for faster recovery than manual reruns.
Tests that passed after a retry were surfaced as warnings, while caching and reducing dependence on unreliable external components were recommended to address flakiness beyond retries.
FAQ
Why did retries improve the theoretical CI pass rate so dramatically?
A single test failure caused by a 0.05% failure probability had to occur twice when the test was retried once. That reduced the calculated per-test failure probability to 0.000025% and raised the full-suite estimate from 35% to 99.95% under the stated independence assumptions.
How did the system handle failures that could not be retried in place?
Failures with side effects were classified as fatal and required the test environment to be reloaded. This was slower than retrying within the same job but faster than waiting for a developer to rerun the job manually.
How did Shopify prevent retries from hiding flaky tests?
The system detected tests that passed after a retry and notified developers. These outcomes were treated as warnings because retries can conceal reliability issues even though they improve CI availability.
Source and freshness
- Source
- shopify.engineering
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.