---
title: "Test Budget: Time Constrained CI Feedback"
description: "Shopify’s core monolith contains more than 170,000 tests, and variable CI feedback times create delays and context switches as the test pool grows. The Test Infrastructure team evaluated a Test Budget approach that builds prioritized orders on top of deterministic test selection, then stops execution at a predetermined time to increase the chance of finding failures early while accepting some risk. Six criteria were compared—failure_rate, avg_duration, churn, coverage, complexity, and default/random—using Time to First Failure, APFD, and Convergence Index; historical results were stored and queried with Redis Sorted Sets. The analysis found failure_rate had the best detection rate, while churn and complexity were inappropriate; running 60% of the suite detected 80% of failures in the mean case, and 70% of the test-selection suite detected 50% at the fifth percentile."
---

# Test Budget: Time Constrained CI Feedback

[Shopify](https://yomu.fyi/company/shopify) · 2023-10-18 · Mar 7, 2022

**Type:** Benchmark

## Summary

Shopify’s core monolith contains more than 170,000 tests, and variable CI feedback times create delays and context switches as the test pool grows. The Test Infrastructure team evaluated a Test Budget approach that builds prioritized orders on top of deterministic test selection, then stops execution at a predetermined time to increase the chance of finding failures early while accepting some risk. Six criteria were compared—failure\_rate, avg\_duration, churn, coverage, complexity, and default/random—using Time to First Failure, APFD, and Convergence Index; historical results were stored and queried with Redis Sorted Sets. The analysis found failure\_rate had the best detection rate, while churn and complexity were inappropriate; running 60% of the suite detected 80% of failures in the mean case, and 70% of the test-selection suite detected 50% at the fifth percentile.

## Context

Shopify’s growing codebase, development team, and test pool increased the time between code check-ins and CI results. Feedback times could vary considerably, causing delayed regression detection and more frequent developer context switches. The analysis examined whether a fixed test-execution time could speed CI by accepting more risk while still finding failures early.

## Approach / What changed

The system applies test prioritization on top of deterministic test selection, using historical results to produce ordered test sets and enforcing a predetermined execution-time budget. Six criteria were evaluated: failure\_rate, avg\_duration, churn, coverage, complexity, and default/random order. Time to First Failure, Average Percentage of Faults Detected, and Convergence Index were used to compare the criteria and choose a constraint. Test-result data was processed through a Rails application and persisted in Redis Sorted Sets.

## Takeaways

- The failure\_rate prioritization criterion had the best failure-detection rate, marginally outperforming the default random order.
- In the mean case, running 60% of the test suite detected 80% of failures with failure\_rate prioritization; default order detected 50%.
- At the fifth percentile, running 70% of the already reduced test-selection suite detected 50% of failures.

**Tags:** [CI/CD](https://yomu.fyi/topic/ci-cd), [Performance](https://yomu.fyi/topic/performance), [Redis](https://yomu.fyi/topic/redis), [Testing](https://yomu.fyi/topic/testing)

- Source: [Shopify](https://shopify.engineering/test-budget-time-constrained-ci-feedback)
- Source URL: https://shopify.engineering/test-budget-time-constrained-ci-feedback
- Ingested by Yomu: 2026-08-30T15:27:29.459Z

[Read original post](https://shopify.engineering/test-budget-time-constrained-ci-feedback)