How we raised mobile end-to-end test stability to 98%

Summary

Shopify’s largest mobile app had become blocked by flaky end-to-end tests, which failed when screens took longer to render and had been removed from pull-request checks. Since 2023, tests used Appium through WebdriverIO and React Native Test IDs, allowing timing shortcuts and assertions about component-tree presence rather than what merchants could see or use. The rebuild hides Appium behind a strict builder-style wrapper that requires an assertion after every action; computer vision uses screenshots, PaddleOCR for text, and OpenCV for icons, with Test IDs as an opt-in fallback. After promotion into blocking CI, stability reached 98% of individual test successes over total runs, up from 50% with the old API, on two platforms. A pre-promotion flakiness gate rejects tests that exceed a failure threshold across repeated runs, while remaining failures largely involve network problems or simulator boot failures.

Context

The Shopify mobile app’s end-to-end suite had become unreliable: screens could take longer to render, tests used timing pauses, and assertions could verify component-tree presence without confirming that a merchant could see or use an element. The resulting flakiness blocked more good pull requests than bad ones, leading to removal of the suite from pull-request checks.

Approach / What changed

The team built an opinionated wrapper around Appium with a strict builder-style API. Every action requires an assertion, reusable step sequences are supported, and escape hatches are explicitly marked UNSAFE_. The framework finds text and icons visually from screenshots using PaddleOCR and OpenCV, keeps Test IDs as an opt-in fallback, emits annotated videos, and runs through one CLI across local, CI, and remote devices. A repeated-run gate checks stability before tests enter blocking CI.

Takeaways

• Every builder step must declare the expected post-action state; assertions are validated to be false before the action and true afterward.
• Visual matching uses PaddleOCR for text and OpenCV for Polaris SVG icons, including grayscale matching, color-inverted variants, size variations, and adjacency rules to disambiguate duplicates.
• Annotated videos show what text or element the test searched for, what it found, and where it tapped, allowing many failures to be diagnosed without a rerun.

Related reading