Selective Test Execution at Stripe: Fast CI for a 50M-line Ruby monorepo

Summary

Stripe’s Ruby monorepo contains about 50 million lines, 100,000 test files, and roughly 1.2 million test units, making sequential execution impractical at an estimated four months. To keep CI within minutes, Selective Test Execution (STE) runs tests whose recorded file accesses intersect files changed since a suitable baseline, averaging about 5% of the suite and under 10% of always-run compute. A dynamically linked C++ library loaded with LD_PRELOAD intercepts file opens, associates them with hierarchical test scopes, and propagates tracing to child processes; logs are aggregated into a selection index outside the syscall hot path. Guardrails cover tests affected by directory discovery, previously failing tests, selective linting, generated files, and reproducible database-backed baselines ordered by Monotonic Revision IDs. The system is presented as a way to preserve safety and affordability while scaling CI for a highly dynamic language.

Context

Stripe needs CI builds for an approximately 50-million-line Ruby monorepo to finish in a few minutes rather than the four months that sequentially running its roughly 1.2 million test units would require. The codebase’s metaprogramming, dynamic dispatch, runtime configuration, and non-Ruby dependencies make perfectly reliable static test dependency analysis impractical.

Approach / What changed

STE dynamically records which files each test accesses. An LD_PRELOAD-loaded C++ shared library intercepts open syscalls, attributes accesses to hierarchical test scopes, propagates through child processes, and writes lightweight per-process logs. A scheduler aggregates these logs into a file-to-scope selection index, while file inventories and database-backed baselines identify changes, including generated files. Guardrails always select certain tests and support selective linting.

Takeaways

• STE uses runtime file-access data to capture dependencies on Ruby files, configuration, fixtures, templates, generated artifacts, and other inputs that static analysis may miss.
• The file-access interceptor keeps the open path lightweight by appending records to per-process temporary files; aggregation, indexing, and inversion happen outside the syscall hot path.
• Tests involving directory globbing or file discovery are marked mandatory, previously failing tests are rerun, and linting tools receive changed-file data so they do not scan the entire repository.

Related reading