---
title: "CI/CD"
description: "152 posts about CI/CD, summarised, each linking to the original."
---

# CI/CD
> 152 posts about CI/CD, summarised, each linking to the original.

## Articles

### [Successfully Merging the Work of 1000+ Developers](https://yomu.fyi/post/successfully-merging-the-work-of-1000-developers.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 14, 2019
- Type: Problem & solution

Shopify’s single monolith supports changes 40 times a day and roughly 400 commits to master daily, making safe trunk-based development difficult at the scale of 1,000+ developers. Its first Merge Queue limited deployment drift and batch size, but queued pull requests did not receive CI, soft conflicts could break master, and a browser extension enabled accidental direct merges. Merge Queue v2 replaces that workflow with GitHub comments and webhooks, programmatic branch protection, an emergency /shipit --emergency path, and a predictive branch where queued changes receive CI before merging. A reconciliation algorithm synchronizes that branch through GitHub’s GraphQL API, while failure-tolerance thresholds handle flaky tests and remove repeatedly failing pull requests. CI starts when requests enter the queue; batches of eight and three concurrently tested batches balance throughput, risk, and resource use, while the changes improved user experience, deployment safety, and deploy throughput.


### [Automated Continuous Deployment at Heroku](https://yomu.fyi/post/automated-continuous-deployment-at-heroku.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Bernerd Schaefer
- Published: Oct 22, 2019
- Type: Problem & solution

Heroku’s Runtime team replaced a manual, coordination-heavy deployment process after team growth and Common Runtime sharding expanded work from one infrequently changed application in two environments to multiple services across 12 environments. Existing Heroku Pipelines deploy merged, CI-passing changes to staging, while the Go-based cedar-service-deployer reconciles differences between pipeline stages and promotes them when a series of safety checks pass. Those checks include a 30-minute age gate, deployment and incident controls, master-branch verification, Librato alerts, and HTTP 200 checks for configured URLs, including platform integration tests. The stateless system can handle concurrent rollouts, and the team added practices such as isolated early stages, feature flags, and separated database-migration changes. After about two years, it managed 23 services, allowing engineers to merge and rely on healthy systems for global release.


### [Living on the Edge of Rails](https://yomu.fyi/post/living-on-the-edge-of-rails.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 21, 2019
- Type: Explainer

Shopify describes running its Core monolith on the unreleased Rails 6 master branch, or “Edge of Rails,” since February 2019, with the goal of continuously using the framework’s latest revision. Instead of annual upgrades, weekly changes reduce diff size and let developers use upstream features quickly, while a 130,000-plus-test suite finds edge cases and informs patches to Rails and related projects such as Zeitwerk. Solid Track automates weekly Rails bumps, opens GitHub pull requests, and triggers git bisect when CI fails to identify the breaking commit. Green builds move through ten-minute canary deployments before reaching the remaining servers. The approach trades the stability of released dependencies for faster fixes and features, relying on monitoring, tests, and controlled deployment to manage risk.


### [Mobile Release Engineering at Scale with Shipit Mobile](https://yomu.fyi/post/mobile-release-engineering-at-scale-with-shipit-mobile.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: May 17, 2019
- Type: Problem & solution

Shopify built Shipit Mobile to replace error-prone, manual mobile app releases with a shared platform for creating, viewing, and managing releases. The process creates a release branch from master, builds a candidate for each commit, increments build numbers, and starts separate CI pipelines for testing and release builds. Shipit Mobile uploads the same binary and metadata to distribution channels including Google Play, the App Store, and internal support access, using store APIs and Fastlane, while Slack notifications make release status visible beyond the release captain. Developers can bypass CI when necessary for emergencies, and configuration favors convention over configuration for project metadata. After more than six months in production, the platform reduced release complexity, shortened the release cadence from three weeks to one week, and helped more people become release captains.


### [Loki, a Dynamic Mock Server for HTTP/TCP Testing](https://yomu.fyi/post/loki-a-dynamic-mock-server-for-http-tcp-testing.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Thuy Nguyen
- Published: Apr 10, 2019
- Type: Problem & solution

Grab built Loki, a dynamic mock server written in Golang that simulates backend services on local developer machines and CI pipelines. Mobile app testing previously suffered from heavy dependencies on complex, brittle staging environments and interconnected services communicating over HTTP, HTTPS, and TCP. Loki handles both HTTP and TCP traffic on distinct ports while exposing a unified RESTful API to manage test expectations. It provides runtime flexibility through sandboxed JavaScript execution, configurable request sequence ordering, and an in-memory cron scheduler for TCP push messages. Adopting Loki decoupled mobile releases from staging stability, improving delivery cycles and enabling automated UI testing with Espresso and XCUITest.


### [The Unreasonable Effectiveness of Test Retries: An Android Monorepo Case Study](https://yomu.fyi/post/the-unreasonable-effectiveness-of-test-retries-an-android-monorepo-cas.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: Alejandro Rodriguez Salamanca
- Published: Jan 8, 2019
- Type: Problem & solution

An Android monorepo expansion and larger team increased the number of tests run on master, exposing how flaky tests and infrastructure failures could erode CI confidence. Under an independence assumption, tests passing 99.95% each produced a theoretical full-suite pass rate of only 35% across 20 pipelines, while retrying each failed test once raised it to 99.95%. Shopify therefore added retries for setup failures, distinguishing failures retriable within a job from fatal failures requiring the test environment to reload, and retried failed tests up to three times. The system also flagged tests that passed after retry as warnings, since retries can hide reliability problems and slow builds. In practice, Android app pipeline pass rate rose from 31% to almost 90% after retry mechanisms and Gradle caching, while iOS rose from 67% to 97%.


### [Mockers - Overcoming Testing Challenges at Grab](https://yomu.fyi/post/mockers-overcoming-testing-challenges-at-grab.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Mayank Gupta
- Published: Sep 18, 2018
- Type: Problem & solution

Grab operates over 250 microservices communicating over HTTP and gRPC, making shared staging environments costly, ambiguous in ownership, and fragile due to inconsistent data and uncoordinated deployments. To address these testing bottlenecks, Grab created Mockers, a Go SDK and CLI tool backed by a central monorepo of mock servers for local-box and CI testing. Mockers automatically generates HTTP and gRPC mock servers from Swagger specifications and protobuf files, returning configured network responses without internal business logic. By incorporating Grab's in-house chaos SDK middleware, Mockers also enables repeatable resiliency and contract testing locally without relying on code-level mocks. While Grab still mandates integration testing on distributed staging environments with live data, Mockers enables developers to detect complex defects and contract mismatches earlier.


### [Mobile Tophatting at Shopify](https://yomu.fyi/post/mobile-tophatting-at-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: Pedro Piñera
- Published: Sep 11, 2018
- Type: Tutorial

Shopify requires developers to test features because it has no QA team, and its manual “tophatting” means running changes locally before merge. Mobile tophatting was slow because switching branches could invalidate Xcode/Android Studio caches, with builds taking up to 15 minutes. The new process skips compilation by exporting CI-built iOS and Android binaries into disposable PR environments, exposing them through the CI API and GitHub commit statuses, then extending the dev command to authenticate, find and filter artifacts, select an app and simulator or emulator, install it, and launch it. Developers can target pull requests, repositories, branches, or build URLs, while iOS apps are zipped .app folders and Android apps retain APK names. The team reports a positive response and lists local artifact caching, real-device support, and simpler commit or branch inputs as improvements.


### [Introducing the Merge Queue](https://yomu.fyi/post/introducing-the-merge-queue.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: Darren Worrall
- Published: Jun 8, 2018
- Type: Announcement

Shopify describes a merge queue added to Shipit to reduce failures in its large monolithic Rails application, where growing pull request volume made manually checking whether master was safe unsustainable. The queue adds pull requests from GitHub, merges them when master is healthy, waits for required CI statuses, and can reject stale branches using configurable divergence thresholds. It is encouraged rather than enforced, while changing the browser extension’s visual emphasis helped make queued merging the default workflow; over 90% of core application pull requests use it. Shipit also handles unsafe deployment commits by allowing explicit marking and automatically detecting reverts of undeployed commits, marking the original and intermediate commits unsafe. These changes let deployment continue safely around reverted work without requiring developers to manually unlock the pipeline.


### [Securing Dependencies for Rails 5.2 Active Storage](https://yomu.fyi/post/securing-dependencies-for-rails-5-2-active-storage.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Craig Ingram
- Published: May 22, 2018
- Type: Problem & solution

Supporting Rails 5.2 Active Storage on Heroku required handling media files for PDF and video previews while addressing FFmpeg’s security and packaging constraints. Ubuntu Trusty’s lack of FFmpeg support, Xenial’s community-maintained package and missing vulnerability backports, and concerns about Libav led the team to build its own binary. Using a fork of sffmpeg and modified CMake configuration, the team created a static FFmpeg build that removed non-free software and unused components such as RTMP and xvid, reducing dependencies and attack surface. Docker-based CircleCI automation compiles the binary, packages it as a Debian archive, uploads it to AWS S3, and lets the buildpack install it without root access. The resulting package is about 30MB instead of roughly 200MB and can be updated in around 20 minutes after upstream security releases.


### [Deep Dive into iOS Automation at Grab - Continuous Delivery](https://yomu.fyi/post/deep-dive-into-ios-automation-at-grab-continuous-delivery.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Sun Xiangxin
- Published: Apr 23, 2017
- Type: Explainer

Grab manages continuous delivery for its iOS applications using four build configurations per target: Adhoc QA, Hot Dogfood, Dogfood, and Testflight. The engineering team moved away from Fastlane in favor of custom bash scripts under 100 lines that execute clean builds and archives via xcodebuild. To eliminate manual logins for build execution, Grab developed a server-side Swift application integrating SlackKit and a bot named Iris. Incoming Slack commands are parsed and scheduled onto a serial DispatchQueue to trigger the appropriate build scripts. Deployment and rollback of updates to Iris and its build scripts are handled through Capistrano.


### [Deep Dive into iOS Automation at Grab - Integration Testing](https://yomu.fyi/post/deep-dive-into-ios-automation-at-grab-integration-testing.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Sun Xiangxin
- Published: Apr 18, 2017
- Type: Problem & solution

Grab replaced third-party cloud CI services with an in-house Xcode Server pipeline to run automated iOS integration tests across an expanding engineering team. Instead of performing fresh repository clones on shared cloud instances, Xcode Server pulls code incrementally and mirrors local developer configurations. The team automated testing bots through Ruby scripts and Arcanist hooks tied to Phabricator code reviews, dynamically provisioning bots on diffs and deleting them after merges. This architecture preserves step-by-step UI test execution logs and screenshots for easier failure diagnostics. To maintain stability on on-premises hardware, pre-integration scripts clear application cache states using simctl commands.


[Newer posts](https://yomu.fyi/topic/ci-cd/page/7.md)