Loading…
Topic
150 posts about CI/CD. Every summary links to the original.
See the idea
A five-step workflow for turning agent output into versioned artifacts: scope a repo per agent, branch instead of copying, run in an isolated sandbox, gate changes with push-triggered evals, then deploy or roll back.
Buildkite splits CI into a hosted control plane and agents you run on your own machines. This comparison weighs that self-hosted agent model against fully hosted runners so you can pick the right fleet.
Anka is a Docker-like macOS virtualization layer for disposable CI environments. This architecture shows how images, Mac hosts, snapshots, build agents, and artifacts connect.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Shopify describes running its Core monolith on the unreleased Rails 6 master branch, or “Edge of Rails,” since February 2019, with the goal of continuously using the framework’s latest revision. Instead of annual upgrades, weekly changes reduce diff size and let developers use upstream features quickly, while a 130,000-plus-test suite finds edge cases and informs patches to Rails and related projects such as Zeitwerk. Solid Track automates weekly Rails bumps, opens GitHub pull requests, and triggers git bisect when CI fails to identify the breaking commit. Green builds move through ten-minute canary deployments before reaching the remaining servers. The approach trades the stability of released dependencies for faster fixes and features, relying on monitoring, tests, and controlled deployment to manage risk.
Shopify built Shipit Mobile to replace error-prone, manual mobile app releases with a shared platform for creating, viewing, and managing releases. The process creates a release branch from master, builds a candidate for each commit, increments build numbers, and starts separate CI pipelines for testing and release builds. Shipit Mobile uploads the same binary and metadata to distribution channels including Google Play, the App Store, and internal support access, using store APIs and Fastlane, while Slack notifications make release status visible beyond the release captain. Developers can bypass CI when necessary for emergencies, and configuration favors convention over configuration for project metadata. After more than six months in production, the platform reduced release complexity, shortened the release cadence from three weeks to one week, and helped more people become release captains.
Grab built Loki, a dynamic mock server written in Golang that simulates backend services on local developer machines and CI pipelines. Mobile app testing previously suffered from heavy dependencies on complex, brittle staging environments and interconnected services communicating over HTTP, HTTPS, and TCP. Loki handles both HTTP and TCP traffic on distinct ports while exposing a unified RESTful API to manage test expectations. It provides runtime flexibility through sandboxed JavaScript execution, configurable request sequence ordering, and an in-memory cron scheduler for TCP push messages. Adopting Loki decoupled mobile releases from staging stability, improving delivery cycles and enabling automated UI testing with Espresso and XCUITest.
An Android monorepo expansion and larger team increased the number of tests run on master, exposing how flaky tests and infrastructure failures could erode CI confidence. Under an independence assumption, tests passing 99.95% each produced a theoretical full-suite pass rate of only 35% across 20 pipelines, while retrying each failed test once raised it to 99.95%. Shopify therefore added retries for setup failures, distinguishing failures retriable within a job from fatal failures requiring the test environment to reload, and retried failed tests up to three times. The system also flagged tests that passed after retry as warnings, since retries can hide reliability problems and slow builds. In practice, Android app pipeline pass rate rose from 31% to almost 90% after retry mechanisms and Gradle caching, while iOS rose from 67% to 97%.
Grab operates over 250 microservices communicating over HTTP and gRPC, making shared staging environments costly, ambiguous in ownership, and fragile due to inconsistent data and uncoordinated deployments. To address these testing bottlenecks, Grab created Mockers, a Go SDK and CLI tool backed by a central monorepo of mock servers for local-box and CI testing. Mockers automatically generates HTTP and gRPC mock servers from Swagger specifications and protobuf files, returning configured network responses without internal business logic. By incorporating Grab's in-house chaos SDK middleware, Mockers also enables repeatable resiliency and contract testing locally without relying on code-level mocks. While Grab still mandates integration testing on distributed staging environments with live data, Mockers enables developers to detect complex defects and contract mismatches earlier.
Shopify requires developers to test features because it has no QA team, and its manual “tophatting” means running changes locally before merge. Mobile tophatting was slow because switching branches could invalidate Xcode/Android Studio caches, with builds taking up to 15 minutes. The new process skips compilation by exporting CI-built iOS and Android binaries into disposable PR environments, exposing them through the CI API and GitHub commit statuses, then extending the dev command to authenticate, find and filter artifacts, select an app and simulator or emulator, install it, and launch it. Developers can target pull requests, repositories, branches, or build URLs, while iOS apps are zipped .app folders and Android apps retain APK names. The team reports a positive response and lists local artifact caching, real-device support, and simpler commit or branch inputs as improvements.
Shopify describes a merge queue added to Shipit to reduce failures in its large monolithic Rails application, where growing pull request volume made manually checking whether master was safe unsustainable. The queue adds pull requests from GitHub, merges them when master is healthy, waits for required CI statuses, and can reject stale branches using configurable divergence thresholds. It is encouraged rather than enforced, while changing the browser extension’s visual emphasis helped make queued merging the default workflow; over 90% of core application pull requests use it. Shipit also handles unsafe deployment commits by allowing explicit marking and automatically detecting reverts of undeployed commits, marking the original and intermediate commits unsafe. These changes let deployment continue safely around reverted work without requiring developers to manually unlock the pipeline.
Supporting Rails 5.2 Active Storage on Heroku required handling media files for PDF and video previews while addressing FFmpeg’s security and packaging constraints. Ubuntu Trusty’s lack of FFmpeg support, Xenial’s community-maintained package and missing vulnerability backports, and concerns about Libav led the team to build its own binary. Using a fork of sffmpeg and modified CMake configuration, the team created a static FFmpeg build that removed non-free software and unused components such as RTMP and xvid, reducing dependencies and attack surface. Docker-based CircleCI automation compiles the binary, packages it as a Debian archive, uploads it to AWS S3, and lets the buildpack install it without root access. The resulting package is about 30MB instead of roughly 200MB and can be updated in around 20 minutes after upstream security releases.
Grab manages continuous delivery for its iOS applications using four build configurations per target: Adhoc QA, Hot Dogfood, Dogfood, and Testflight. The engineering team moved away from Fastlane in favor of custom bash scripts under 100 lines that execute clean builds and archives via xcodebuild. To eliminate manual logins for build execution, Grab developed a server-side Swift application integrating SlackKit and a bot named Iris. Incoming Slack commands are parsed and scheduled onto a serial DispatchQueue to trigger the appropriate build scripts. Deployment and rollback of updates to Iris and its build scripts are handled through Capistrano.
Grab replaced third-party cloud CI services with an in-house Xcode Server pipeline to run automated iOS integration tests across an expanding engineering team. Instead of performing fresh repository clones on shared cloud instances, Xcode Server pulls code incrementally and mirrors local developer configurations. The team automated testing bots through Ruby scripts and Arcanist hooks tied to Phabricator code reviews, dynamically provisioning bots on diffs and deleting them after merges. This architecture preserves step-by-step UI test execution logs and screenshots for easier failure diagnostics. To maintain stability on on-premises hardware, pre-integration scripts clear application cache states using simctl commands.
The best engineering reads, once a week.
