Test Budget: Time Constrained CI Feedback

Summary

Shopify’s core monolith contains more than 170,000 tests, and variable CI feedback times create delays and context switches as the test pool grows. The Test Infrastructure team evaluated a Test Budget approach that builds prioritized orders on top of deterministic test selection, then stops execution at a predetermined time to increase the chance of finding failures early while accepting some risk. Six criteria were compared—failure_rate, avg_duration, churn, coverage, complexity, and default/random—using Time to First Failure, APFD, and Convergence Index; historical results were stored and queried with Redis Sorted Sets. The analysis found failure_rate had the best detection rate, while churn and complexity were inappropriate; running 60% of the suite detected 80% of failures in the mean case, and 70% of the test-selection suite detected 50% at the fifth percentile.

Context

Shopify’s growing codebase, development team, and test pool increased the time between code check-ins and CI results. Feedback times could vary considerably, causing delayed regression detection and more frequent developer context switches. The analysis examined whether a fixed test-execution time could speed CI by accepting more risk while still finding failures early.

Approach / What changed

The system applies test prioritization on top of deterministic test selection, using historical results to produce ordered test sets and enforcing a predetermined execution-time budget. Six criteria were evaluated: failure_rate, avg_duration, churn, coverage, complexity, and default/random order. Time to First Failure, Average Percentage of Faults Detected, and Convergence Index were used to compare the criteria and choose a constraint. Test-result data was processed through a Rails application and persisted in Redis Sorted Sets.

Takeaways

• The failure_rate prioritization criterion had the best failure-detection rate, marginally outperforming the default random order.
• In the mean case, running 60% of the test suite detected 80% of failures with failure_rate prioritization; default order detected 50%.
• At the fifth percentile, running 70% of the already reduced test-selection suite detected 50% of failures.

Related reading