Loading…
Topic
151 posts about CI/CD. Every summary links to the original.
See the idea
A five-step workflow for turning agent output into versioned artifacts: scope a repo per agent, branch instead of copying, run in an isolated sandbox, gate changes with push-triggered evals, then deploy or roll back.
Buildkite splits CI into a hosted control plane and agents you run on your own machines. This comparison weighs that self-hosted agent model against fully hosted runners so you can pick the right fleet.
Anka is a Docker-like macOS virtualization layer for disposable CI environments. This architecture shows how images, Mac hosts, snapshots, build agents, and artifacts connect.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Grab's 69.3 GiB multi-module Go monorepo caused commands like go get to take over 18 minutes as Git repeatedly traversed commit history, downloaded large worktrees, and overloaded their GitLab VCS infrastructure. To bypass direct VCS queries without losing automatic updates for external repositories, the team deployed the Athens Go module proxy configured in fallback network mode. They used the GOVCS environment variable to disable Git access specifically for the monorepo path, forcing Athens to fall back to its internal object storage when resolving monorepo modules. A dedicated CI pipeline pre-populates and refreshes the Athens cache whenever new monorepo modules are released. This setup reduced monorepo go get execution times to approximately 12 seconds and allowed a 70% scale-down of the Athens proxy cluster.
Upgrading to Xcode 13.1 introduced severe CI test instability and high CPU utilisation for Grab's iOS development team. To address this, the team integrated observability tools across their UITest pipeline to pinpoint performance bottlenecks and test flakiness. Interventions included isolating spotlight.app to curb CPU spikes, replacing Safari with a mock browser for deep link tests, and booting simulators with pre-granted permissions. The team also built custom network tracking tools to enforce resource mocking and transitioned tests away from arbitrary sleep commands to explicit wait wrappers. These combined changes cut CI runtimes, decreased CPU utilisation by over 50%, and stabilized automated test executions.
Stateful stream processing frameworks like Apache Flink present unique deployment challenges because conventional canary and blue-green strategies can cause data inaccuracies or state divergence. Grab's real-time data platform team encountered risks of state loss, manual rollback overhead, and absent health checks in their Kubernetes and Spinnaker deployment pipeline. To resolve these operational issues, the team redesigned the deployment workflow around automated Flink savepointing and programmatic health monitoring. The new pipeline halts existing applications after capturing state snapshots and Kafka offsets, monitors target deployments via API health probes, and executes automated rollbacks using versioned ConfigMaps and replica metadata annotations. This automated process ensures state consistency during upgrades and eliminates manual intervention during deployment failures.
In 2021, GitHub released OpenID Connect support for GitHub Actions to let workflows retrieve short-lived cloud session tokens instead of storing static secrets. Tinder Security Labs discovered widespread IAM role misconfigurations that allow external repositories to assume target roles and access AWS resources. The primary issue stems from IAM trust policies that authenticate the GitHub identity provider but omit strict validation on the subject claim. Researchers located exposed role identifiers via public workflow searches and confirmed that arbitrary external GitHub Actions workflows could assume them, including roles within AWS official repositories. Remediation requires configuring trust policies to restrict the audience and validate repository namespaces, while unauthorized assumptions can be detected in CloudTrail logs.
As Grab expanded its superapp, manual QA sign-offs led to prolonged testing cycles and a peak in major production defects around 2019. To counter defect leakage and improve delivery pace, the engineering organization transitioned to a shift-left testing strategy across its software development lifecycle. Quality engineers began writing Given/When/Then acceptance tests prior to coding so that software engineers could execute them during development, accompanied by Definition of Ready and Definition of Done standards. The team also structured a multi-layer test strategy spanning unit tests, pipeline-executed UI component tests with mocked APIs, backend integration tests, and a custom internal end-to-end framework. Between 2019 and 2022, these changes reduced major and critical production issues by 60% and development-phase critical bugs by 40%.
Grab's real-time data platform team transitioned from an Atlantis-driven Terraform workflow to an in-house GitOps platform called Khone to manage streaming infrastructure resources like Kafka topics and Flink pipelines. The earlier setup suffered from coarse-grained access controls, required manual merge request comments, and lacked flexible validation capabilities within native configuration files. Khone derives environment parameters directly from standardized directory paths and uses Python with the python-hcl2 library to inspect and validate resource definitions before executing Terraform stages in parallel. To prevent configuration tampering in merge requests, CI/CD pipeline definitions and execution scripts are isolated in a separate administrative repository and fetched during job runs using shallow Git clones.
Internal documentation is presented as a productivity and scaling tool for engineering organizations, especially as people move between teams, projects, or companies and remote work makes writing a default channel for knowledge sharing. Shopify’s approach combines standardized writing practices with three primary locations: Markdown microsites built with Docusaurus, internal GitHub for issues, READMEs, setup guidance, and searchable code, and the Vault, a GitHub- and Markdown-powered source of contextual information. The article argues that documentation should be treated as a product: it should be searchable, user-friendly, kept fresh, integrated into CI/CD, and hosted where engineers already work. It also recommends making writing routine through small contributions, paired writing, peer review, and audience-aware context, concluding that accessible documentation speeds onboarding, reduces duplicated problem-solving, supports asynchronous and cross-team collaboration, and improves developer literacy.
Shopify needed automated GitHub workflows that could clone from private repositories, update organization-level projects, and trigger downstream workflows but could not use GITHUB_TOKEN. Personal access tokens supplied those capabilities, yet introduced developer churn, user-attributed actions, and no organization-enforced rotation or expiry. The proposed system uses GitHub Actions and a GitHub App to generate repository secrets that rotate automatically while preserving PAT-like references in existing workflows. Its serverless design delegates scheduling, managed infrastructure, approvals, secret management, auditing, and break-glass operation to GitHub, while a custom TypeScript action encrypts and writes generated secrets. Prototyping exposed that parallel one-second jobs could accrue multiple billable minutes, prompting sequential execution in a consolidated workflow; the stated goal is improved resilience to developer churn and reduced leaked-secret blast radius.
The November 7–14, 2022 logbook presents updates intended to make Elixir applications easier to deploy and monitor in production, alongside documentation and storage guidance. Mark’s contribution explains how to set up Github Actions for Elixir CI, including the work needed to automate deployment workflows. A separate guide uses OpenTelemetry to monitor a production Elixir deployment and trace an N+1 query performance problem, while the Django section reports community progress toward more comprehensive Fly documentation. The logbook also points to a MinIO deployment guide for self-hosted, S3-compatible object storage, noting that colocating storage in the same datacenter as an application can matter although using an S3 host such as AWS S3 is described as easier in some cases.
Supabase CLI v1 is moving out of beta with capabilities for managing database migrations and generating TypeScript types, alongside a beta Management API. The CLI can manage organizations and projects, link projects locally, run schema and manual migration workflows, support CI/CD through local and GitHub Actions workflows, and generate types from local or remote databases. Migration changes include improved manually written migrations, an experimental migra-backed db diff that is faster and produces more concise DDL, and local branching for testing DDL without data loss; the source notes that generated scripts are not perfect. The REST-based Management API, previously called the Admin API, is consumed by the CLI and exposes programmatic management of organizations, projects, Edge Functions, and more, with endpoints intended for CI/CD and test-environment creation.
The article examines how a CTO can remain technically credible after moving from the individual contributor track into leadership. It argues that leadership inevitably reduces technical depth because performance reviews, one-on-ones, organizational communication, and other people responsibilities limit time in code, making the transition an intentional choice for those who value managing people. Its recommended practices include voracious reading, coding as a hobby, participating in hackathons, learning directly from more technical colleagues, maintaining a private network of technical advisors, and aggressively clearing bugs, incident-remediation work, and backlog. The author illustrates hands-on learning with Shopify’s Spin environment, which provisions a cloud development instance and opens a repository in about 30 seconds, and concludes that controlling operational issues preserves time for innovation and technical leadership.
Supabase has released an updated version of its framework-specific Auth Helpers libraries, introducing official support for SvelteKit alongside existing React and Next.js tooling to simplify server-side rendering authentication. The updated libraries transition from the @supabase/supabase-auth-helpers namespace to @supabase/auth-helpers within a dedicated monorepo managed by Turborepo. Structuring the codebase with Turborepo and changesets enables publishing isolated packages per framework through GitHub Actions, ensuring each helper contains only its own relevant dependencies. Versioning has also been reset to sub-0.x to reflect early framework lifecycles like pre-1.0 SvelteKit and manage expectations for future breaking changes. Meanwhile, the team has documented migration steps from previous packages and announced that Remix Auth Helpers are currently in active development.
Shopify describes how a bug bounty report led to application fixes and Rails contributions aimed at making open-source software safer. The 2021 Shopify Experiments report showed that a leaked development-environment secret could be used to generate trusted-looking data and reach remote code execution in a local development environment. The root cause was ActiveSupport’s MessageEncryptor using Ruby’s unsafe Marshal serializer by default in Rails versions up to 7.0.X, allowing untrusted data to trigger deserialization risks. Shopify first configured JSON in its application, then helped change the default serializers for MessageEncryptor and MessageVerifier to JSON in Rails 7.1.X, with an upgrade path for existing applications. It also developed a patch that monitors Marshal load and dump calls, raises on unexpected gem-originated calls, and runs in local development and continuous integration to prevent new usage.
Grab's Cauldron Machine Learning Platform team managed continuous delivery across multiple pipelines by using nested GitLab CI configurations, but soon encountered platform limitations such as the 100-include ceiling and bloated 1,800-line YAML files. To address these constraints, the team implemented GitLab Dynamic Child Pipelines to programmatically create execution stages on the fly. They built a command-line utility in Rust that runs git diff against the base branch, extracts pipeline and stage metadata using configurable stop patterns, and applies a template to produce a dynamic CI definition. On the master branch, the tool fetches the diff artifact from the source branch through the GitLab API to execute identical generation logic. This architectural shift reduced the root CI file from 1,800 lines down to a constant 50 lines, allowing configuration size to remain stable regardless of repository growth.
Shopify’s core monolith contains more than 170,000 tests, and variable CI feedback times create delays and context switches as the test pool grows. The Test Infrastructure team evaluated a Test Budget approach that builds prioritized orders on top of deterministic test selection, then stops execution at a predetermined time to increase the chance of finding failures early while accepting some risk. Six criteria were compared—failure_rate, avg_duration, churn, coverage, complexity, and default/random—using Time to First Failure, APFD, and Convergence Index; historical results were stored and queried with Redis Sorted Sets. The analysis found failure_rate had the best detection rate, while churn and complexity were inappropriate; running 60% of the suite detected 80% of failures in the mean case, and 70% of the test-selection suite detected 50% at the fifth percentile.
RubyConf 2021: The Talks You Might Have Missed compiles talks presented by Shopify’s Ruby and Rails infrastructure engineers at the November conference in Denver, offering ways to revisit sessions for attendees who missed them. The program covers Ruby implementation topics including compiler history, Aaron Patterson’s pure-Ruby JIT compiler, YJIT’s incremental integration inside CRuby, parser generators, variable-width allocation, and Ruby memory layout. Other sessions address gradual Sorbet adoption, native extension compilation and security, pair programming, memoization performance, executable code as data, Ruby archaeology, fast CI, and Ractor-based parallel testing. The roundup reports early YJIT performance results, describes a CI environment with 2.8 million Rails monolith lines and 210,000 Ruby tests, and notes variable-width allocation passed Shopify’s Rails CI while serving more than 500 million requests in a week of limited production traffic.
Shopify identified a dependency confusion vulnerability through its bug bounty program; an attacker could exploit ambiguous package sources to install a malicious dependency and gain access to local, CI/CD, and production environments. To assess exposure, the Ruby Conventions team collected Bundler versions from repositories in a data lake and found around 600 susceptible Ruby applications. An experimental Bundler plugin converted Gemfile.lock files to the explicit-source format while restoring existing gem versions, and a CI-backed CLI later provided per-repository environments, parallel execution, pull requests, and easier failure tracking. After incremental migrations exposed configuration and tooling issues, Shopify updated supporting systems, set Bundler requirements, changed local and CI defaults, and worked with Bundler maintainers; the company reports migrating all its applications in less than a day and contributing to Bundler 2.2.21 and 2.2.22.
Shopify’s year-end roundup curates ten favorite engineering posts from 2021, spanning Ruby, storefronts, data analysis, databases, search, GraphQL, CI, APIs, and mobile development, while also noting several older reader favorites. The selection includes tutorials on building a web app with standard Ruby libraries, exploratory data analysis, GraphQL, Apache Beam for search, and React Native App Clips, alongside technical accounts of Hydrogen, YJIT, MySQL upgrades, and GraphQL rate limiting. YJIT was merged upstream and included in Ruby 3.1.0, while the CI team reduced the p95 of Shopify’s core monolith from 45 minutes to 18 minutes. The roundup also describes query-cost-based GraphQL rate limiting, rollback-testing roadblocks during a MySQL upgrade, and an App Clip effort that used a fail-fast, iterative approach.
Recent findings from the 2021 State of AI Report and Kaggle State of Machine Learning and Data Science Survey indicate that machine learning is expanding into critical infrastructure while Transformers become general-purpose architectures across text, vision, and audio. In response, organizations face questions about scaling infrastructure, team composition, and engineering maturity. Rather than treating machine learning as isolated sandbox experiments or hiring solely data scientists, teams benefit from adopting established software engineering and DevOps principles like versioning, testing, automation, and continuous deployment. Furthermore, the rise of pre-trained Transformer architectures enables practitioners to fine-tune existing off-the-shelf models rather than training from scratch, reducing compute costs and training duration. Tools from platforms such as Hugging Face streamline model deployment, latency optimization, and infrastructure abstraction.
YJIT’s benchmarking site combines open-source benchmark runners with Jekyll, GitHub Pages, and GitHub Actions to publish frequently refreshed results. The author wanted to compare Ruby configurations, expand reporting beyond console output, and avoid recurring operational work. Benchmarks run on an AWS instance rather than in GitHub Actions, while Actions schedules jobs, uses repository secrets for SSH, and commits generated data back to the repository; Jekyll builds the Pages site from that data. Reports include D3.js and Ruby-generated SVG graphs, tables, textual summaries, and JSON tripwire data that can trigger an issue when performance drops too far. The reported results show YJIT 20.6% faster than interpreted CRuby and 17.4% faster than MJIT, with Railsbench gains of 18.5% and 21.0% respectively.
The best engineering reads, once a week.
