How we raised mobile end-to-end test stability to 98%

Summary

Shopify’s largest mobile app had become blocked by flaky end-to-end tests, which failed when screens took longer to render and had been removed from pull-request checks. Since 2023, tests used Appium through WebdriverIO and React Native Test IDs, allowing timing shortcuts and assertions about component-tree presence rather than what merchants could see or use. The rebuild hides Appium behind a strict builder-style wrapper that requires an assertion after every action; computer vision uses screenshots, PaddleOCR for text, and OpenCV for icons, with Test IDs as an opt-in fallback. After promotion into blocking CI, stability reached 98% of individual test successes over total runs, up from 50% with the old API, on two platforms. A pre-promotion flakiness gate rejects tests that exceed a failure threshold across repeated runs, while remaining failures largely involve network problems or simulator boot failures.

Takeaways

1. 1

   Every builder step must declare the expected post-action state; assertions are validated to be false before the action and true afterward.

2. 2

   Visual matching uses PaddleOCR for text and OpenCV for Polaris SVG icons, including grayscale matching, color-inverted variants, size variations, and adjacency rules to disambiguate duplicates.

3. 3

   Annotated videos show what text or element the test searched for, what it found, and where it tapped, allowing many failures to be diagnosed without a rerun.