---
title: "Keeping Developers Happy with a Fast CI"
description: "Shopify’s Test Infrastructure team tackled a 45-minute 95th-percentile CI time for its Core monolith, which had more than 170,000 tests and ran on every git push. They instrumented Buildkite commands and prioritized agent preparation, dependency building, and test execution using command frequency and duration. Disk I/O from writing large caches slowed Docker startup, so the team increased disk capacity and write speed, mounted most caches read-only, skipped unchanged database and asset work through hashes, parallelized setup, and expanded changed-file test mappings. These changes reduced container-start p95 from 90 to 25 seconds, raised builds avoiding the full suite from 45% to over 60%, improved test stability from 88% to 97%, and lowered overall CI p95 to 18 minutes."
---

# Keeping Developers Happy with a Fast CI

[Shopify](https://yomu.fyi/company/shopify) · 2023-10-18 · Feb 24, 2021

**Type:** Problem & solution

## Summary

Shopify’s Test Infrastructure team tackled a 45-minute 95th-percentile CI time for its Core monolith, which had more than 170,000 tests and ran on every git push. They instrumented Buildkite commands and prioritized agent preparation, dependency building, and test execution using command frequency and duration. Disk I/O from writing large caches slowed Docker startup, so the team increased disk capacity and write speed, mounted most caches read-only, skipped unchanged database and asset work through hashes, parallelized setup, and expanded changed-file test mappings. These changes reduced container-start p95 from 90 to 25 seconds, raised builds avoiding the full suite from 45% to over 60%, improved test stability from 88% to 97%, and lowered overall CI p95 to 18 minutes.

## Context

Shopify’s Core monolith CI ran on every git push and had a 95th-percentile build time of around 45 minutes. With more than 170,000 tests, developers were frustrated by delays, while the test suite continued growing by 20–30% annually. The team set an objective of bringing CI p95 below 10 minutes without increasing cloud computing expenses by more than 10%.

## Approach / What changed

The team instrumented every CI command, analyzed execution time and frequency, and addressed bottlenecks in agent preparation, dependency building, and testing. Changes included increasing disk capacity and write speed, sharing read-only caches, hashing database and asset inputs to skip unchanged work, running setup steps in parallel, expanding changed-file test mappings, and temporarily removing tests that caused severe delays.

## Takeaways

- Disk I/O, rather than Docker itself, caused slow container startup after large cache downloads; increasing disk capacity and write speed and sharing read-only caches reduced container-start p95 from 90 to 25 seconds.
- Hashing database structure and migration inputs allowed unchanged Rails database setup to be skipped, while similar asset checks and parallel execution reduced one dependency job from five minutes to about three.
- Test mappings for fixtures and other changed files increased builds that avoided the full suite from 45% to over 60%, improved test stability from 88% to 97%, and contributed to an overall CI p95 of 18 minutes.

**Tags:** [CI/CD](https://yomu.fyi/topic/ci-cd), [Docker](https://yomu.fyi/topic/docker), [Performance](https://yomu.fyi/topic/performance), [Testing](https://yomu.fyi/topic/testing)

- Source: [Shopify](https://shopify.engineering/faster-shopify-ci)
- Source URL: https://shopify.engineering/faster-shopify-ci
- Ingested by Yomu: 2026-08-31T01:10:13.650Z

[Read original post](https://shopify.engineering/faster-shopify-ci)