---
title: "How we raised mobile end-to-end test stability to 98%"
description: "Shopify’s largest mobile app had become blocked by flaky end-to-end tests, which failed when screens took longer to render and had been removed from pull-request checks. Since 2023, tests used Appium through WebdriverIO and React Native Test IDs, allowing timing shortcuts and assertions about component-tree presence rather than what merchants could see or use. The rebuild hides Appium behind a strict builder-style wrapper that requires an assertion after every action; computer vision uses screenshots, PaddleOCR for text, and OpenCV for icons, with Test IDs as an opt-in fallback. After promotion into blocking CI, stability reached 98% of individual test successes over total runs, up from 50% with the old API, on two platforms. A pre-promotion flakiness gate rejects tests that exceed a failure threshold across repeated runs, while remaining failures largely involve network problems or simulator boot failures."
---

# How we raised mobile end-to-end test stability to 98%

[Shopify](https://yomu.fyi/company/shopify) · 2023-10-18 · Aug 12, 2026

**Type:** Problem & solution

## Summary

Shopify’s largest mobile app had become blocked by flaky end-to-end tests, which failed when screens took longer to render and had been removed from pull-request checks. Since 2023, tests used Appium through WebdriverIO and React Native Test IDs, allowing timing shortcuts and assertions about component-tree presence rather than what merchants could see or use. The rebuild hides Appium behind a strict builder-style wrapper that requires an assertion after every action; computer vision uses screenshots, PaddleOCR for text, and OpenCV for icons, with Test IDs as an opt-in fallback. After promotion into blocking CI, stability reached 98% of individual test successes over total runs, up from 50% with the old API, on two platforms. A pre-promotion flakiness gate rejects tests that exceed a failure threshold across repeated runs, while remaining failures largely involve network problems or simulator boot failures.

## Context

The Shopify mobile app’s end-to-end suite had become unreliable: screens could take longer to render, tests used timing pauses, and assertions could verify component-tree presence without confirming that a merchant could see or use an element. The resulting flakiness blocked more good pull requests than bad ones, leading to removal of the suite from pull-request checks.

## Approach / What changed

The team built an opinionated wrapper around Appium with a strict builder-style API. Every action requires an assertion, reusable step sequences are supported, and escape hatches are explicitly marked UNSAFE\_. The framework finds text and icons visually from screenshots using PaddleOCR and OpenCV, keeps Test IDs as an opt-in fallback, emits annotated videos, and runs through one CLI across local, CI, and remote devices. A repeated-run gate checks stability before tests enter blocking CI.

## Takeaways

- Every builder step must declare the expected post-action state; assertions are validated to be false before the action and true afterward.
- Visual matching uses PaddleOCR for text and OpenCV for Polaris SVG icons, including grayscale matching, color-inverted variants, size variations, and adjacency rules to disambiguate duplicates.
- Annotated videos show what text or element the test searched for, what it found, and where it tapped, allowing many failures to be diagnosed without a rerun.

**Tags:** [CI/CD](https://yomu.fyi/topic/ci-cd), [Developer Experience](https://yomu.fyi/topic/developer-experience), [Reliability](https://yomu.fyi/topic/reliability), [Testing](https://yomu.fyi/topic/testing)

- Source: [Shopify](https://shopify.engineering/mobile-e2e-testing)
- Source URL: https://shopify.engineering/mobile-e2e-testing
- Ingested by Yomu: 2026-08-30T13:17:14.807Z

[Read original post](https://shopify.engineering/mobile-e2e-testing)