Headless Screenshot Testing in Thumbtack’s Android Apps

Summary

Thumbtack required a faster, emulator-free method to execute over 1,500 Android screenshot tests during continuous integration. Previously, running UI tests on physical devices or emulators proved too slow for code review verification jobs, causing tests to run post-merge and allowing regressions to persist. After evaluating Paparazzi and Google's experimental Compose Preview Testing, the team selected Roborazzi, which runs on top of Robolectric's native graphics engine within a standard JVM. This headless setup accommodates custom Activities, Dagger field injection, and legacy View systems while executing alongside standard unit tests. Moving screenshot tests to the unit test level enabled gating merges on passing tests, colocating tests inside feature modules, and increasing total CI job runtime by only twenty percent.

Context

Thumbtack maintained over 1,500 Android UI screenshot tests that originally required physical devices or emulators to execute drawing operations. Because emulator execution was too slow, these tests could not run during Jenkins code review verification jobs. Consequently, screenshot tests ran post-merge, leaving UI test failures unresolved for days in the main branch.

Approach / What changed

The team adopted Roborazzi, a headless screenshot testing library built on Robolectric's native graphics framework, enabling tests to run in a plain JVM alongside unit tests. They rejected Paparazzi because it lacked support for custom Activity classes required for Dagger field injection in legacy Views, and bypassed Compose Preview Testing due to bugs and incompatibility with legacy View System code.

Takeaways

• Running screenshot tests in a plain JVM via Robolectric native graphics enabled over 1,500 UI tests to gate pull requests directly in Jenkins verification jobs.
• Paparazzi was ruled out because Thumbtack's legacy code relied on Dagger field injection requiring custom Activity and Application classes.
• Host platform rendering discrepancies between UNIX and MacOS in Robolectric native graphics were resolved by introducing a small pixel shift tolerance during image comparisons.

Related reading

• Thumbtack · Aug 8, 2026

  How We Moved 1,500 Android Screenshot Tests to Roborazzi

  Thumbtack migrated 1,500 Android screenshot tests from Firebase Test Lab to Roborazzi to eliminate slow and error-prone testing on remote emulators. The engineering team ported test suites from androidTest to test in batches of 20, adjusting test rules and annotations for Robolectric. Because Roborazzi runs in a single JVM process with parallel execution, the team had to debug shared font cache pollution and cross-platform shadow rendering inconsistencies between macOS and Debian. They resolved configuration repetition by mapping Robolectric annotations into module-level robolectric.properties files and avoided stale cached images by clearing Roborazzi's cache via a custom Gradle task. The migration significantly decreased CI runtimes while making local test recording and verification easier.

  Zachary Wander

• Sentry · Jun 11, 2026

  Catch visual regressions with Snapshots, now in beta