Loading…
Topic
150 posts about CI/CD. Every summary links to the original.
See the idea
A five-step workflow for turning agent output into versioned artifacts: scope a repo per agent, branch instead of copying, run in an isolated sandbox, gate changes with push-triggered evals, then deploy or roll back.
Buildkite splits CI into a hosted control plane and agents you run on your own machines. This comparison weighs that self-hosted agent model against fully hosted runners so you can pick the right fleet.
Anka is a Docker-like macOS virtualization layer for disposable CI environments. This architecture shows how images, Mac hosts, snapshots, build agents, and artifacts connect.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Developer Infrastructure at Shopify spans local and cloud development environments, CI systems, deploys, and technology-specific foundations for web and mobile applications. The post models these teams against a three-phase develop–validate–deploy workflow, distinguishing horizontally integrated teams that own one phase from vertically integrated teams whose systems span phases for a particular technology. It uses this model to identify dependencies, gaps, differing scopes, collaboration patterns, and three work avenues—concentration, expansion, and interfacing—with the mission of improving and scaling the cycle. The analysis informed organizational changes, including a Client Foundations team, closer collaboration, and plans for workflow-wide user research focused on the developer journey from writing a patch to production.
A May 19, 2021 Shipit! event brought together Colin Gray, Haris Mahmood, Guil Varandas, and Michelle Fernandez, developers setting Shopify’s React Native standards, to discuss writing performant apps. The Q&A covers accessibility and design guidance through Restyle and Polaris documentation; their responsive layouts avoid fixed sizing where possible and use breakpoints for fonts and spacing. They describe React Native state as the usual source of truth while native modules manage internal state and send updates through EventEmitter, with SQLite in Point of Sale as an exception that still has one source of truth. Jest handles unit and integration tests, Buildkite manages CI builds, nightly builds are automatically distributed to testers, and failing feature-branch tests block merges to main.
Conveyor, an in-house continuous delivery system at Grab, introduces hermetic deployments by tracking application code alongside static and dynamic configuration parameters. This hermeticity guarantees that production releases use combinations of versioned artifacts previously verified in staging, preventing rollback incompatibilities. Conveyor replaces single multi-environment pipelines with decoupled pipelines, while automating cluster locking, deployment slot scheduling, release note generation, and canary monitoring with automated rollbacks. These automations reduced production deployment failure rates from 1.5% to an average of 0.3% over a three-month period. Grab also doubled the volume of production changes between 2018 and 2020 while saving more than 5,000 man-days of engineering effort in 2020.
Shopify’s Test Infrastructure team tackled a 45-minute 95th-percentile CI time for its Core monolith, which had more than 170,000 tests and ran on every git push. They instrumented Buildkite commands and prioritized agent preparation, dependency building, and test execution using command frequency and duration. Disk I/O from writing large caches slowed Docker startup, so the team increased disk capacity and write speed, mounted most caches read-only, skipped unchanged database and asset work through hashes, parallelized setup, and expanded changed-file test mappings. These changes reduced container-start p95 from 90 to 25 seconds, raised builds avoiding the full suite from 45% to over 60%, improved test stability from 88% to 97%, and lowered overall CI p95 to 18 minutes.
Grab's mobile superapp scaled past 2.5 million lines of code across both Android and iOS, leading to unsustainable local and CI build times under Gradle and Xcode. To address these bottlenecks, the engineering team analyzed their dependency trees and introduced an internal tool to calculate and optimize the build critical path. They also deployed a Kubernetes-autoscaled remote build system using Mainframer for Android and implemented Test Impact Analysis to run only affected tests in pre-merge validation. While dependency decoupling yielded modest 7% to 10% gains and iOS remote builds proved unscalable on Apple hardware, Android remote builds reduced local compile times by up to 50%, and targeted test execution reduced pre-merge pipeline durations by more than 30%.
Shopify’s Starscream data platform handled 76,000 jobs and wrote 300 terabytes daily, but its PySpark workflow was slow and cumbersome for commonplace reporting. An audit found that about 70 percent of PySpark jobs were full batch queries that did not require generalized computing, while users often wrote SQL before translating it into Python. The team built Seamster with dbt and Google BigQuery, adding source and model structures, ownership metadata, Python-based unit tests with fixed mock inputs, CI validation, and warehouse rules. A base layer isolates raw-source changes, while models are organized into base, application-ready, and presentation layers and defined through SQL, schema, README, and test files. After a multiweek beta, most users reported shipping models in days rather than weeks, although dbt’s incremental support could not safely handle some late-arriving-data, key-resolution, rebuild, and very large-model cases.
Grab's passenger Android superapp relies on more than 170 in-house and open-source libraries, incorporating five to ten library bumps into each weekly release. Although developers often avoided updates due to the fear of leaking defects or crashes into production, the engineering team established a risk-assessment framework based on codebase usage and the volume of incoming changes. To minimize update diffs and avoid accumulating large issues, libraries are updated incrementally on a weekly cadence matching upstream release schedules. Automated UI test cases written in Gherkin syntax execute on CI for every merge request, while high-risk bumps trigger targeted QA manual testing. This structured updating process prevents defect leakage while keeping the application compliant with evolving Google Play target API requirements.
Around the end of 2018, Grab's backend architecture consisted of roughly 270 services managed through fragmented, manual deployment workflows. Engineers copied release parameters between build logs, wiki pages, Slack bots, and multiple Jenkins jobs, leading to high operational friction and an average of 10 business days between production updates for a service. To streamline delivery, Grab built Conveyor, an internal automation platform built on top of open-source Spinnaker. Conveyor introduced a custom user interface focused on pipeline visibility and a pipeline-as-code DSL called Artificer using Jsonnet files in the monorepository. The platform automatically registers build artifacts with commit metadata to eliminate manual parameter entry and automatically provisions integration, staging, and production pipelines.
Shopify’s tophatting process manually verifies coworkers’ changes before pull requests are approved, but React Native reviews previously required saving work, switching branches, rebuilding, and loading the app. The new tool stores compressed JavaScript bundles and their assets in cloud storage when Shopify Build’s CI pipeline processes a React Native pull request. A backend service records metadata such as the app ID, commit SHA, and branch, generates signed URLs, and posts a pull-request comment with a QR code, bundle ID, and download link. A React Native component library handles deep links through a separate Tophat screen, confirms bundle metadata, downloads and decompresses the selected bundle, saves metadata locally, and restarts the app with it. The first iteration lets developers load peer changes within seconds, and almost all Shopify React Native projects now use the tool.
Shopify’s Point Of Sale (POS) app is mission-critical for brick-and-mortar merchants, because downtime can create long lineups, unhappy customers, and lost sales. Unlike server-side systems, mobile fixes depend on Apple and Google review, and merchants may delay installing approved updates, so the team focuses on preventing bugs and shortening recovery time. Its practices include layered automated and manual testing, component ownership with automatic Code Owners reviews, automated mobile “tophatting,” and weekly releases supported by a rotating Release Captain and ShipIt Mobile. Staged rollouts, beta flags, minimum-version enforcement for rare critical security fixes, and dedicated Primary and Secondary ATCs limit exposure and clarify incident response. ChatOps-based incident channels capture findings for root cause analysis, while rotating on-call responsibilities reduce context switching; these measures helped the app scale to larger merchants while shipping faster and improving reliability.
Managing dependencies in a multi-module monorepo created developer friction at Grab due to unexpected changes from previous vendoring attempts and accidental imports. Because Go modules were not yet enabled directly for builds, the team implemented a continuous integration check that executes go mod vendor and rejects merge requests if any diffs exist in go.mod or the vendor directory. Adopting this CI check required configuring SSH deploy keys for private repositories, adding retry logic for network-related false positives, and standardizing on a single Go version to prevent checksum discrepancies. To streamline ongoing maintenance across hundreds of dependencies, the team developed an automated tool named AutoVend Bot. The bot runs go list -m -u all to detect updates and opens a scheduled batch of merge requests each day for human review.
Engineering and data teams across Grab originally operated independent Apache Airflow instances, causing duplicate maintenance overhead and frequent job failures around scaling, logging, and dependency management. To resolve this, a dedicated team developed a centralized orchestration platform that runs isolated, containerized Airflow instances per team on Amazon EKS. The platform categorizes deployments into three size tiers and provisions dedicated Redis brokers, RDS metadata stores, and Vault secret sidecars using Terraform and custom Helm charts. Teams customize container images using shared GitLab CI/CD templates, while worker scaling is handled via Kubernetes Horizontal Pod Autoscalers. Today, the platform runs roughly 20 Airflow instances executing between 1,000 and 60,000 daily jobs per instance.
The post presents a hands-on tutorial for Python developers moving a local Django application into production on Heroku. It frames the work around 12-factor practices and covers repository hygiene, modular settings, continuous delivery stages, static-asset middleware, environment configuration, database provisioning, deployment, scaling, and troubleshooting. The configuration changes include excluding virtual environments, caches, SQLite data, and generated static files; moving settings.py to settings/base.py; updating DJANGO_SETTINGS_MODULE; adding Whitenoise; and supplying SECRET_KEY and a Heroku Postgres add-on. The deployment uses Git, the Heroku CLI, git push heroku main, and heroku ps:scale web=1, with heroku logs --tail and heroku run bash for diagnosis. The walkthrough concludes with the app deployed and accessible through heroku open.
Shopify’s monolithic Ruby repository has more than 150,000 tests, grows 20–30% annually, and takes 30–40 minutes to run across hundreds of Docker containers, making required full-suite CI costly and vulnerable to intermittent failures. The team built a dynamic-analysis test-selection system that logs method calls during each test, records files in each call graph, and maps changed files to relevant tests. Because Ruby, Rails metaprogramming, and non-Ruby files complicate tracing, the system adds Rails patches and fallback rules, runs extra tests when mappings lag, and executes the full suite asynchronously on every deploy. After two months and 8,360 merged commits, it achieved 99.94% failure recall, selected about 60% of tests, and reduced compute time by roughly 25%. Developers requested full-suite runs on fewer than 2% of pull requests.
Shopify describes its release culture as a combination of beliefs and behaviors shaping how developers ship software, with trust, velocity, and safeguards as stated goals. The team measures developer experience through biannual happiness surveys, shipping Slack channels, dogfooding, internal support, and UX research that includes shadowing developers and testing prototypes with interns and new hires. Its release pipeline runs from a pull request and /shipit command through the merge queue, Master, a Canary environment receiving 5% of incoming requests, and Production after a 10-minute testing period without intervention or automated alerts. Developers own releases without managers, sign-offs, or release windows, while /shipit --emergency provides a single fast recovery path. Shopify also uses Beta Flags for gradual rollouts and measures /shipit success through survey feedback and average pull-request time-to-production.
Parallel test execution in Xcode can suffer from test time imbalance when tasks finish at significantly different times across parallel simulator workers. Analysis of Xcode scheduling logs shows that the runner groups tests by test class and dispatches all tests from the same class to a single simulator. Attempts to customize the suite by swizzling XCTestSuite fail because made-up suites initialize only after tests are dispatched. To overcome this grouping constraint, unique tokens or test names are appended to the class name component in `-only-testing` command-line arguments. This trick forces Xcode to treat each test as an independent class, successfully distributing individual tests across separate workers.
Shopify describes the challenge of scaling mobile development without slowing feature delivery as teams and projects adopt divergent tools, making knowledge sharing, cross-project contribution, and incident response harder. It extends ServicesDB, previously used to track production services, to treat mobile apps as services with owners and explicit expectations encoded in configuration files. Automatic checks cover ownership, contact information, CI and Shipit Mobile deployment configuration, internal-tool versioning, and production error and crash monitoring. Failed checks generate GitHub issues with remediation instructions, while the contract helps the Mobile Tooling team identify compatible projects and helps developers understand why tooling may not work. The implementation currently has basic checks, with planned work including mobile-app service-level objectives, easier project creation, and shorter build times.
Terrier is an open-source tool for identifying and verifying files and other components in OCI images and running containers, introduced alongside a Blackhat Europe talk on exploiting builds in the cloud. It targets supply-chain and forensic use cases, such as checking whether an image contains a known file or malicious package. YAML configuration selects image or container mode, supplies SHA256 hashes and, for container checks, paths; Terrier scans image layers or a container’s merged folder while keeping processing in memory for performance. Examples locate /usr/local/go/bin/go, report that the Jeilyfish package is absent, and verify curl and Go files in a running container; CircleCI and GitHub Actions examples show how checks can gate image-build pipelines.
Shopify’s single monolith supports changes 40 times a day and roughly 400 commits to master daily, making safe trunk-based development difficult at the scale of 1,000+ developers. Its first Merge Queue limited deployment drift and batch size, but queued pull requests did not receive CI, soft conflicts could break master, and a browser extension enabled accidental direct merges. Merge Queue v2 replaces that workflow with GitHub comments and webhooks, programmatic branch protection, an emergency /shipit --emergency path, and a predictive branch where queued changes receive CI before merging. A reconciliation algorithm synchronizes that branch through GitHub’s GraphQL API, while failure-tolerance thresholds handle flaky tests and remove repeatedly failing pull requests. CI starts when requests enter the queue; batches of eight and three concurrently tested batches balance throughput, risk, and resource use, while the changes improved user experience, deployment safety, and deploy throughput.
Heroku’s Runtime team replaced a manual, coordination-heavy deployment process after team growth and Common Runtime sharding expanded work from one infrequently changed application in two environments to multiple services across 12 environments. Existing Heroku Pipelines deploy merged, CI-passing changes to staging, while the Go-based cedar-service-deployer reconciles differences between pipeline stages and promotes them when a series of safety checks pass. Those checks include a 30-minute age gate, deployment and incident controls, master-branch verification, Librato alerts, and HTTP 200 checks for configured URLs, including platform integration tests. The stateless system can handle concurrent rollouts, and the team added practices such as isolated early stages, feature flags, and separated database-migration changes. After about two years, it managed 23 services, allowing engineers to merge and rely on healthy systems for global release.
The best engineering reads, once a week.
