---
title: "Spark Joy by Running Fewer Tests"
description: "Shopify’s monolithic Ruby repository has more than 150,000 tests, grows 20–30% annually, and takes 30–40 minutes to run across hundreds of Docker containers, making required full-suite CI costly and vulnerable to intermittent failures. The team built a dynamic-analysis test-selection system that logs method calls during each test, records files in each call graph, and maps changed files to relevant tests. Because Ruby, Rails metaprogramming, and non-Ruby files complicate tracing, the system adds Rails patches and fallback rules, runs extra tests when mappings lag, and executes the full suite asynchronously on every deploy. After two months and 8,360 merged commits, it achieved 99.94% failure recall, selected about 60% of tests, and reduced compute time by roughly 25%. Developers requested full-suite runs on fewer than 2% of pull requests."
---

# Spark Joy by Running Fewer Tests

[Shopify](https://yomu.fyi/company/shopify) · 2023-10-18 · Jun 11, 2020

**Type:** Problem & solution

## Summary

Shopify’s monolithic Ruby repository has more than 150,000 tests, grows 20–30% annually, and takes 30–40 minutes to run across hundreds of Docker containers, making required full-suite CI costly and vulnerable to intermittent failures. The team built a dynamic-analysis test-selection system that logs method calls during each test, records files in each call graph, and maps changed files to relevant tests. Because Ruby, Rails metaprogramming, and non-Ruby files complicate tracing, the system adds Rails patches and fallback rules, runs extra tests when mappings lag, and executes the full suite asynchronously on every deploy. After two months and 8,360 merged commits, it achieved 99.94% failure recall, selected about 60% of tests, and reduced compute time by roughly 25%. Developers requested full-suite runs on fewer than 2% of pull requests.

## Context

Shopify’s monolithic repository contains over 150,000 tests, grows by 20–30% annually, and requires 30–40 minutes to run across hundreds of Docker containers. Requiring every pull request to pass the full suite creates waiting or context-switching costs, while intermittent failures increase retries and reduce CI stability.

## Approach / What changed

The team uses dynamic analysis to log method calls during tests, build call graphs, and map files to tests. Changed files determine the selected tests. Rails patches trace some non-Ruby files, fallback rules cover metaprogramming and stale mappings, and the full suite runs asynchronously on every deployed commit.

## Takeaways

- The system achieved 99.94% failure recall after two months, missing five failing tests among 8,360 merged commits.
- About 60% of tests are selected overall, and roughly 40% of builds run fewer than 20% of the full test suite.
- Selective execution saves about 25% of compute time, although dynamic-analysis runs and container, database, and cache setup limit the total savings.

**Tags:** [CI/CD](https://yomu.fyi/topic/ci-cd), [Performance](https://yomu.fyi/topic/performance), [Ruby](https://yomu.fyi/topic/ruby), [Testing](https://yomu.fyi/topic/testing)

- Source: [Shopify](https://shopify.engineering/spark-joy-by-running-fewer-tests)
- Source URL: https://shopify.engineering/spark-joy-by-running-fewer-tests
- Ingested by Yomu: 2026-08-31T01:12:29.906Z

[Read original post](https://shopify.engineering/spark-joy-by-running-fewer-tests)