---
title: "CI/CD"
description: "152 posts about CI/CD, summarised, each linking to the original."
---

# CI/CD
> 152 posts about CI/CD, summarised, each linking to the original.

## Articles

### [GitHub Does My Operations Homework: A Ruby Speed Story](https://yomu.fyi/post/github-does-my-operations-homework-a-ruby-speed-story.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 19, 2021
- Type: Tutorial

YJIT’s benchmarking site combines open-source benchmark runners with Jekyll, GitHub Pages, and GitHub Actions to publish frequently refreshed results. The author wanted to compare Ruby configurations, expand reporting beyond console output, and avoid recurring operational work. Benchmarks run on an AWS instance rather than in GitHub Actions, while Actions schedules jobs, uses repository secrets for SSH, and commits generated data back to the repository; Jekyll builds the Pages site from that data. Reports include D3.js and Ruby-generated SVG graphs, tables, textual summaries, and JSON tripwire data that can trigger an issue when performance drops too far. The reported results show YJIT 20.6% faster than interpreted CRuby and 17.4% faster than MJIT, with Railsbench gains of 18.5% and 21.0% respectively.


### [Reusing Code with React Native Packages at Shopify](https://yomu.fyi/post/reusing-code-with-react-native-packages-at-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Sep 29, 2021
- Type: Explainer

Shopify’s React Native teams needed to reuse shared functionality across Shop, Inbox, Point of Sale, Shopify Mobile, and Local Delivery without requiring each team to build and maintain separate package infrastructure. The React Native Foundations team chose a monorepo containing npm packages, a fixture app, shared root configuration, generated CI, and an internal documentation site, with code generation for package bootstrapping. Packages use TypeScript project references, Yarn Workspaces, Lerna publishing, and custom Babel, Metro, and Podfile configuration to support cross-package references, hot reload, and local native dependencies. Dependency graphs and file-change triggers limit CI work to affected packages and dependents, while independent versioning and release scripts manage publication; since June 2020, 17 packages have been developed, including 10 contributed by product teams, although release automation and coordination across multiple monorepos remain future improvements.


### [Modelling Developer Infrastructure Teams](https://yomu.fyi/post/modelling-developer-infrastructure-teams.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 13, 2021
- Type: Explainer

Developer Infrastructure at Shopify spans local and cloud development environments, CI systems, deploys, and technology-specific foundations for web and mobile applications. The post models these teams against a three-phase develop–validate–deploy workflow, distinguishing horizontally integrated teams that own one phase from vertically integrated teams whose systems span phases for a particular technology. It uses this model to identify dependencies, gaps, differing scopes, collaboration patterns, and three work avenues—concentration, expansion, and interfacing—with the mission of improving and scaling the cycle. The analysis informed organizational changes, including a Client Foundations team, closer collaboration, and plans for workflow-wide user research focused on the developer journey from writing a patch to production.


### [Shipit! Presents: How We Write React Native Apps](https://yomu.fyi/post/shipit-presents-how-we-write-react-native-apps.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: May 27, 2021
- Type: Announcement

A May 19, 2021 Shipit! event brought together Colin Gray, Haris Mahmood, Guil Varandas, and Michelle Fernandez, developers setting Shopify’s React Native standards, to discuss writing performant apps. The Q&A covers accessibility and design guidance through Restyle and Polaris documentation; their responsive layouts avoid fixed sizing where possible and use breakpoints for fonts and spacing. They describe React Native state as the usual source of truth while native modules manage internal state and send updates through EventEmitter, with SQLite in Point of Sale as an exception that still has one source of truth. Jest handles unit and integration tests, Buildkite manages CI builds, nightly builds are automatically distributed to testers, and failing feature-branch tests block merges to main.


### [Our Journey to Continuous Delivery at Grab (Part 2)](https://yomu.fyi/post/our-journey-to-continuous-delivery-at-grab-part-2.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Sylvain Bougerel
- Published: May 10, 2021
- Type: Problem & solution

Conveyor, an in-house continuous delivery system at Grab, introduces hermetic deployments by tracking application code alongside static and dynamic configuration parameters. This hermeticity guarantees that production releases use combinations of versioned artifacts previously verified in staging, preventing rollback incompatibilities. Conveyor replaces single multi-environment pipelines with decoupled pipelines, while automating cluster locking, deployment slot scheduling, release note generation, and canary monitoring with automated rollbacks. These automations reduced production deployment failure rates from 1.5% to an average of 0.3% over a three-month period. Grab also doubled the volume of production changes between 2018 and 2020 while saving more than 5,000 man-days of engineering effort in 2020.


### [Keeping Developers Happy with a Fast CI](https://yomu.fyi/post/keeping-developers-happy-with-a-fast-ci.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Feb 24, 2021
- Type: Problem & solution

Shopify’s Test Infrastructure team tackled a 45-minute 95th-percentile CI time for its Core monolith, which had more than 170,000 tests and ran on every git push. They instrumented Buildkite commands and prioritized agent preparation, dependency building, and test execution using command frequency and duration. Disk I/O from writing large caches slowed Docker startup, so the team increased disk capacity and write speed, mounted most caches read-only, skipped unchanged database and asset work through hashes, parallelized setup, and expanded changed-file test mappings. These changes reduced container-start p95 from 90 to 25 seconds, raised builds avoiding the full suite from 45% to over 60%, improved test stability from 88% to 97%, and lowered overall CI p95 to 18 minutes.


### [How Grab is Blazing Through the Superapp Bazel Migration](https://yomu.fyi/post/how-grab-is-blazing-through-the-superapp-bazel-migration.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Sergii Grechukha
- Published: Dec 3, 2020
- Type: Problem & solution

Grab's mobile superapp scaled past 2.5 million lines of code across both Android and iOS, leading to unsustainable local and CI build times under Gradle and Xcode. To address these bottlenecks, the engineering team analyzed their dependency trees and introduced an internal tool to calculate and optimize the build critical path. They also deployed a Kubernetes-autoscaled remote build system using Mainframer for Android and implemented Test Impact Analysis to run only affected tests in pre-merge validation. While dependency decoupling yielded modest 7% to 10% gains and iOS remote builds proved unscalable on Apple hardware, Android remote builds reduced local compile times by up to 50%, and targeted test execution reduced pre-merge pipeline durations by more than 30%.


### [How to Build a Production Grade Workflow with SQL Modelling](https://yomu.fyi/post/how-to-build-a-production-grade-workflow-with-sql-modelling.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 19, 2020
- Type: Problem & solution

Shopify’s Starscream data platform handled 76,000 jobs and wrote 300 terabytes daily, but its PySpark workflow was slow and cumbersome for commonplace reporting. An audit found that about 70 percent of PySpark jobs were full batch queries that did not require generalized computing, while users often wrote SQL before translating it into Python. The team built Seamster with dbt and Google BigQuery, adding source and model structures, ownership metadata, Python-based unit tests with fixed mock inputs, CI validation, and warehouse rules. A base layer isolates raw-source changes, while models are organized into base, application-ready, and presentation layers and defined through SQL, schema, README, and test files. After a multiweek beta, most users reported shipping models in days rather than weeks, although dbt’s incremental support could not safely handle some late-arriving-data, key-resolution, rebuild, and very large-model cases.


### [Keeping 170 Libraries Up to Date on a Large Scale Android App](https://yomu.fyi/post/keeping-170-libraries-up-to-date-on-a-large-scale-android-app.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Lucas Nelaupe
- Published: Oct 30, 2020
- Type: Problem & solution

Grab's passenger Android superapp relies on more than 170 in-house and open-source libraries, incorporating five to ten library bumps into each weekly release. Although developers often avoided updates due to the fear of leaking defects or crashes into production, the engineering team established a risk-assessment framework based on codebase usage and the volume of incoming changes. To minimize update diffs and avoid accumulating large issues, libraries are updated incrementally on a weekly cadence matching upstream release schedules. Automated UI test cases written in Gherkin syntax execute on CI for every merge request, while high-risk bumps trigger targeted QA manual testing. This structured updating process prevents defect leakage while keeping the application compliant with evolving Google Play target API requirements.


### [Our Journey to Continuous Delivery at Grab (Part 1)](https://yomu.fyi/post/our-journey-to-continuous-delivery-at-grab-part-1.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Sylvain Bougerel
- Published: Sep 23, 2020
- Type: Problem & solution

Around the end of 2018, Grab's backend architecture consisted of roughly 270 services managed through fragmented, manual deployment workflows. Engineers copied release parameters between build logs, wiki pages, Slack bots, and multiple Jenkins jobs, leading to high operational friction and an average of 10 business days between production updates for a service. To streamline delivery, Grab built Conveyor, an internal automation platform built on top of open-source Spinnaker. Conveyor introduced a custom user interface focused on pipeline visibility and a pipeline-as-code DSL called Artificer using Jsonnet files in the monorepository. The platform automatically registers build artifacts with commit metadata to eliminate manual parameter entry and automatically provisions integration, staging, and production pipelines.


### [Tophatting in React Native](https://yomu.fyi/post/tophatting-in-react-native.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Sep 9, 2020
- Type: Tutorial

Shopify’s tophatting process manually verifies coworkers’ changes before pull requests are approved, but React Native reviews previously required saving work, switching branches, rebuilding, and loading the app. The new tool stores compressed JavaScript bundles and their assets in cloud storage when Shopify Build’s CI pipeline processes a React Native pull request. A backend service records metadata such as the app ID, commit SHA, and branch, generates signed URLs, and posts a pull-request comment with a QR code, bundle ID, and download link. A React Native component library handles deep links through a separate Tophat screen, confirms bundle metadata, downloads and decompresses the selected bundle, saves metadata locally, and restarts the app with it. The first iteration lets developers load peer changes within seconds, and almost all Shopify React Native projects now use the tool.


### [Building Reliable Mobile Applications](https://yomu.fyi/post/building-reliable-mobile-applications.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 14, 2020
- Type: Explainer

Shopify’s Point Of Sale (POS) app is mission-critical for brick-and-mortar merchants, because downtime can create long lineups, unhappy customers, and lost sales. Unlike server-side systems, mobile fixes depend on Apple and Google review, and merchants may delay installing approved updates, so the team focuses on preventing bugs and shortening recovery time. Its practices include layered automated and manual testing, component ownership with automatic Code Owners reviews, automated mobile “tophatting,” and weekly releases supported by a rotating Release Captain and ShipIt Mobile. Staged rollouts, beta flags, minimum-version enforcement for rare critical security fixes, and dedicated Primary and Secondary ATCs limit exposure and clarify incident response. ChatOps-based incident channels capture findings for root cause analysis, while rotating on-call responsibilities reduce context switching; these measures helped the app scale to larger merchants while shipping faster and improving reliability.


### [Go Modules- A Guide for monorepos (Part 2)](https://yomu.fyi/post/go-modules-a-guide-for-monorepos-part-2.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Michael Cartmell
- Published: Aug 12, 2020
- Type: Problem & solution

Managing dependencies in a multi-module monorepo created developer friction at Grab due to unexpected changes from previous vendoring attempts and accidental imports. Because Go modules were not yet enabled directly for builds, the team implemented a continuous integration check that executes go mod vendor and rejects merge requests if any diffs exist in go.mod or the vendor directory. Adopting this CI check required configuring SSH deploy keys for private repositories, adding retry logic for network-related false positives, and standardizing on a single Go version to prevent checksum discrepancies. To streamline ongoing maintenance across hundreds of dependencies, the team developed an automated tool named AutoVend Bot. The bot runs go list -m -u all to detect updates and opens a scheduled batch of merge requests each day for human review.


### [The Journey of Deploying Apache Airflow at Grab](https://yomu.fyi/post/the-journey-of-deploying-apache-airflow-at-grab.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Chandulal Kavar
- Published: Jul 14, 2020
- Type: Problem & solution

Engineering and data teams across Grab originally operated independent Apache Airflow instances, causing duplicate maintenance overhead and frequent job failures around scaling, logging, and dependency management. To resolve this, a dedicated team developed a centralized orchestration platform that runs isolated, containerized Airflow instances per team on Amazon EKS. The platform categorizes deployments into three size tiers and provisions dedicated Redis brokers, RDS metadata stores, and Vault secret sidecars using Terraform and custom Helm charts. Teams customize container images using shared GitLab CI/CD templates, while worker scaling is handled via Kubernetes Horizontal Pod Autoscalers. Today, the platform runs roughly 20 Airflow instances executing between 1,000 and 60,000 daily jobs per instance.


### [From Project to Productionized with Python](https://yomu.fyi/post/from-project-to-productionized-with-python.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Ed Morley
- Published: Jun 22, 2020
- Type: Tutorial

The post presents a hands-on tutorial for Python developers moving a local Django application into production on Heroku. It frames the work around 12-factor practices and covers repository hygiene, modular settings, continuous delivery stages, static-asset middleware, environment configuration, database provisioning, deployment, scaling, and troubleshooting. The configuration changes include excluding virtual environments, caches, SQLite data, and generated static files; moving settings.py to settings/base.py; updating DJANGO\_SETTINGS\_MODULE; adding Whitenoise; and supplying SECRET\_KEY and a Heroku Postgres add-on. The deployment uses Git, the Heroku CLI, git push heroku main, and heroku ps:scale web=1, with heroku logs --tail and heroku run bash for diagnosis. The walkthrough concludes with the app deployed and accessible through heroku open.


### [Spark Joy by Running Fewer Tests](https://yomu.fyi/post/spark-joy-by-running-fewer-tests.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 11, 2020
- Type: Problem & solution

Shopify’s monolithic Ruby repository has more than 150,000 tests, grows 20–30% annually, and takes 30–40 minutes to run across hundreds of Docker containers, making required full-suite CI costly and vulnerable to intermittent failures. The team built a dynamic-analysis test-selection system that logs method calls during each test, records files in each call graph, and maps changed files to relevant tests. Because Ruby, Rails metaprogramming, and non-Ruby files complicate tracing, the system adds Rails patches and fallback rules, runs extra tests when mappings lag, and executes the full suite asynchronously on every deploy. After two months and 8,360 merged commits, it achieved 99.94% failure recall, selected about 60% of tests, and reduced compute time by roughly 25%. Developers requested full-suite runs on fewer than 2% of pull requests.


### [Software Release Culture at Shopify](https://yomu.fyi/post/software-release-culture-at-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Apr 15, 2020
- Type: Explainer

Shopify describes its release culture as a combination of beliefs and behaviors shaping how developers ship software, with trust, velocity, and safeguards as stated goals. The team measures developer experience through biannual happiness surveys, shipping Slack channels, dogfooding, internal support, and UX research that includes shadowing developers and testing prototypes with interns and new hires. Its release pipeline runs from a pull request and /shipit command through the merge queue, Master, a Canary environment receiving 5% of incoming requests, and Production after a 10-minute testing period without intervention or automated alerts. Developers own releases without managers, sign-offs, or release windows, while /shipit --emergency provides a single fast recovery path. Shopify also uses Beta Flags for gradual rollouts and measures /shipit success through survey feedback and average pull-request time-to-production.


### [Tackling UI Test Execution Time Imbalance for Xcode Parallel Testing](https://yomu.fyi/post/tackling-ui-test-execution-time-imbalance-for-xcode-parallel-testing.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Ngoc Thuyen Trinh
- Published: Mar 16, 2020
- Type: Problem & solution

Parallel test execution in Xcode can suffer from test time imbalance when tasks finish at significantly different times across parallel simulator workers. Analysis of Xcode scheduling logs shows that the runner groups tests by test class and dispatches all tests from the same class to a single simulator. Attempts to customize the suite by swizzling XCTestSuite fail because made-up suites initialize only after tests are dispatched. To overcome this grouping constraint, unique tokens or test names are appended to the class name component in \`-only-testing\` command-line arguments. This trick forces Xcode to treat each test as an independent class, successfully distributing individual tests across separate workers.


### [Scaling Mobile Development by Treating Apps as Services](https://yomu.fyi/post/scaling-mobile-development-by-treating-apps-as-services.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jan 16, 2020
- Type: Problem & solution

Shopify describes the challenge of scaling mobile development without slowing feature delivery as teams and projects adopt divergent tools, making knowledge sharing, cross-project contribution, and incident response harder. It extends ServicesDB, previously used to track production services, to treat mobile apps as services with owners and explicit expectations encoded in configuration files. Automatic checks cover ownership, contact information, CI and Shipit Mobile deployment configuration, internal-tool versioning, and production error and crash monitoring. Failed checks generate GitHub issues with remediation instructions, while the contract helps the Mobile Tooling team identify compatible projects and helps developers understand why tooling may not work. The implementation currently has basic checks, with planned work including mobile-app service-level objectives, easier project creation, and shorter build times.


### [Terrier: Open-Source Tool to Identify and Analyze Container and Image Components](https://yomu.fyi/post/terrier-open-source-tool-to-identify-and-analyze-container-and-image-c.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: chris le roy
- Published: Jan 15, 2020
- Type: Tutorial

Terrier is an open-source tool for identifying and verifying files and other components in OCI images and running containers, introduced alongside a Blackhat Europe talk on exploiting builds in the cloud. It targets supply-chain and forensic use cases, such as checking whether an image contains a known file or malicious package. YAML configuration selects image or container mode, supplies SHA256 hashes and, for container checks, paths; Terrier scans image layers or a container’s merged folder while keeping processing in memory for performance. Examples locate /usr/local/go/bin/go, report that the Jeilyfish package is absent, and verify curl and Go files in a running container; CircleCI and GitHub Actions examples show how checks can gate image-build pipelines.


[Newer posts](https://yomu.fyi/topic/ci-cd/page/6.md) · [Older posts](https://yomu.fyi/topic/ci-cd/page/8.md)