How We Moved 1,500 Android Screenshot Tests to Roborazzi

Summary

Thumbtack migrated 1,500 Android screenshot tests from Firebase Test Lab to Roborazzi to eliminate slow and error-prone testing on remote emulators. The engineering team ported test suites from androidTest to test in batches of 20, adjusting test rules and annotations for Robolectric. Because Roborazzi runs in a single JVM process with parallel execution, the team had to debug shared font cache pollution and cross-platform shadow rendering inconsistencies between macOS and Debian. They resolved configuration repetition by mapping Robolectric annotations into module-level robolectric.properties files and avoided stale cached images by clearing Roborazzi's cache via a custom Gradle task. The migration significantly decreased CI runtimes while making local test recording and verification easier.

Context

Thumbtack relied on Firebase Test Lab to capture and compare Jetpack Compose layouts for Android apps, but using remote emulators and devices proved slow, tedious, and error-prone.

Approach / What changed

The team ported 1,500 test suites in batches of 20 from androidTest to test using Roborazzi and Robolectric. They hoisted flags to disable shadows individually, consolidated shared configurations into robolectric.properties files, and hooked a Gradle cache-clearing task into AbstractTestTask.

Takeaways

• Roborazzi executes tests in a single JVM process in parallel, removing emulator process isolation and risking cross-suite state leakage like shared font cache pollution.
• Cross-platform graphics rendering between macOS workstations and Debian CI environments caused shadow discrepancies, which Thumbtack resolved by hoisting flags to disable shadows across layouts.
• Robolectric class annotations can be mapped to camelCase property keys in a robolectric.properties file to standardize configurations such as SDK version, qualifiers, and native graphics mode.

Related reading

• Thumbtack · Mar 31, 2026

  Headless Screenshot Testing in Thumbtack’s Android Apps

  Thumbtack required a faster, emulator-free method to execute over 1,500 Android screenshot tests during continuous integration. Previously, running UI tests on physical devices or emulators proved too slow for code review verification jobs, causing tests to run post-merge and allowing regressions to persist. After evaluating Paparazzi and Google's experimental Compose Preview Testing, the team selected Roborazzi, which runs on top of Robolectric's native graphics engine within a standard JVM. This headless setup accommodates custom Activities, Dagger field injection, and legacy View systems while executing alongside standard unit tests. Moving screenshot tests to the unit test level enabled gating merges on passing tests, colocating tests inside feature modules, and increasing total CI job runtime by only twenty percent.

  Brian Terczynski

• Thumbtack · Apr 21, 2026

  Leveraging AI to build a faster iOS CI system