Loading…
CI/CD
34 posts about CI/CD. Every summary links to the original.
Everything we launched during Agents Week
Agents Week introduced a series of products and primitives across Cloudflare to support an Agentic Internet where autonomous software and humans interact. Developing capable agents requires addressing technical challenges spanning identity, communication, orchestration, memory, observability, and security. Cloudflare introduced the Agent Development Lifecycle alongside tools like Cloudflare Agents for live tracing, replay, and human-in-the-loop approvals. The releases also included programmable CI/CD pipelines capable of repairing failures, Cloudflare Wallets for agent transactions, and Kitesurf, an agent-first browser running in V8 isolates. New protocols and tools like WebMCP, MCPv2, and Cloudflare AI Search aim to make web data and applications readable, discoverable, and searchable by autonomous software.
Shelley JonesGithub ·
Turn one giant AI-generated pull request to a reviewable stack
Coding agents often consolidate full-stack features into monolithic pull requests spanning thousands of lines, resulting in delayed reviews and lower feedback quality. To resolve this bottleneck, engineers can decompose comprehensive features into ordered, single-concern layers using GitHub stacked pull requests and the gh-stack CLI extension. Distinct specialized agents handle individual branches—ranging from foundational data access modules up to client interfaces—while CI checks continuously evaluate each layer against the stack base. When reviewers request upstream modifications, running gh stack sync executes a cascading rebase across all dependent branches, preserving commit signatures and keeping the entire pull request chain mergeable.
Julia MuiruriGithub ·
Tame Dependabot: Group your updates, slow the cadence, keep security fast
Dependabot often floods repository maintainers with individual daily pull requests for single patch bumps, wasting review and continuous integration resources. Microsoft's GCToolkit project mitigated this issue by updating its dependabot.yml configuration to group dependency updates using wildcard patterns and slowing the schedule interval to monthly. The project also expanded coverage to include Maven alongside GitHub Actions so all relevant package ecosystems receive managed updates. Because Dependabot processes security alerts independently of regular version schedules and enforces a default three-day package cooldown, critical vulnerability fixes remain fast while routine maintenance noise drops significantly.
Bruno BorgesSlack ·
Shipyard: How We Built Slack’s Next-Generation EC2 Platform
Slack developed Shipyard to modernize its EC2 infrastructure by replacing continuously updated, mutable instances with immutable deployable artifacts. The platform adopts a layered image architecture starting with a standardized base image called slack-zero, which is built using AWS Image Builder and consumed by downstream service-specific machine images. Rather than running periodic Chef jobs in the background, configuration management is restricted to image baking and initial instance provisioning phases. Fleet management is handled through the Peekaboo inventory system, metric-driven progressive rollouts via Gondola, and automated instance rotation managed by a lifecycle component known as the Reaper. Developers validate cookbook updates prior to merging using Ship Quick, which provisions real infrastructure across dedicated worker fleets to verify provisioning workflows.
Archie GunasekaraGithub ·
Automating cross-repo documentation with GitHub Agentic Workflows
Maintaining documentation across separate repositories often leads to severe lag because technical writers must reverse-engineer shipped features weeks after release. To address this in the Aspire project, the team implemented an automated pipeline using GitHub Agentic Workflows to bridge the product and documentation repositories. When product pull requests merge, a bash step maps milestones to docs release branches before an LLM agent evaluates the diff, drafts documentation updates, and emits structured pull request intents. A dedicated safe-outputs handler materializes these drafts via a scoped GitHub App and assigns the original code reviewers to verify accuracy. Across 396 product pull requests, the system generated 82 documentation pull requests that all merged with a median turnaround time of 44.8 hours.
David PineGrab ·
Scaling out Distroless adoption With AI
Grab is transitioning its microservices to Distroless base images to eliminate unnecessary binaries and reduce vulnerability risks, but the migration risks runtime failures from missing shared objects and system utilities. To safely validate container execution in continuous integration without staging dependencies, the team relied on medium tests that run containerized services alongside internal dependencies managed by Testcontainers. Because hundreds of services lacked this test harness, Grab implemented an agentic workflow using Claude Code and Model Context Protocol integrations to inspect repositories, generate test boilerplate, and resolve configuration errors. Once test baselines are established, an automated patch-test-compare pipeline updates Dockerfiles, constructs multi-stage builds for necessary dynamic libraries, and creates draft merge requests for human approval.
Jia Yee ChongGrab ·
From decentralized Docs-as-Code to a centralized repository: Evolving Grab's documentation strategy
Grab originally adopted a decentralized Docs-as-Code workflow to manage Markdown documentation alongside service code, but the model strained as engineering scaled. Dispersed repositories led to fragmented terminology, uncoordinated repository moves, uneven linting standards, and noisy search results. To resolve these issues, the organization transitioned to a centralized documentation repository with standardized templates, automated CI quality gates, and a unified search overlay combining fuzzy title search with Glean. Consolidating Markdown into a single repository also established a clean, structured knowledge base tailored for internal AI agents. The centralized setup fostered continuous maintenance, averaging 27 daily commits and an update merged roughly every 50 minutes.
Karen KueDropbox ·
Beyond code generation: rethinking engineering productivity in the age of AI agents
Dropbox shares how widespread AI code generation shifts software development bottlenecks downstream into code review, CI infrastructure, and validation pipelines. To adapt, they built Nova, an internal coding agent platform that safely automates scoped tasks such as migrations and flaky test remediation. They also evolved their developer productivity framework to measure end-to-end customer impact and code quality rather than simple pull request throughput.
Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki OkumuraDropbox ·
Introducing Nova, our internal platform for coding agents
Dropbox developed Nova, an internal platform that runs AI coding agents in isolated cloud environments integrated with their Bazel monorepo. The platform supports both interactive developer workflows and autonomous background tasks, such as automated CI debugging, flaky test remediation, and codebase-wide migrations. By pairing code generation with automated validation and strict execution guardrails, Nova ensures generated fixes are tested and reproducible.
Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura,Mike White,Kevin AltschulerDropbox ·
Reducing our monorepo size to improve developer velocity
Dropbox reduced its core server monorepo size from 87GB to 20GB after approaching GitHub's 100GB limit and suffering from hour-long clone times. The excessive bloat was traced to Git's delta compression heuristics interacting poorly with internationalization directory paths, generating inefficient diffs across different languages. By collaborating with GitHub to run an aggressive server-side repack with tuned window and depth settings, Dropbox cut clone times to under 15 minutes.
Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura,Mike White,Kevin Altschuler,Facundo Agriel,Ishan MishraGrab ·
Enabling R8 optimization at scale with AI-assisted debugging
Grab experienced widespread Application Not Responding spikes across its Android superapp, driven by memory pressure and complex Jetpack Compose layouts embedded in legacy code. While switching to advanced R8 optimization promised significant performance gains, obfuscated stack traces and two-hour remote compilation cycles stalled investigation across nine million lines of code. To resolve this, engineers built Model Context Protocol tools to automate APK decompilation, deobfuscation, and code context extraction. The team paired these tools with an AI workflow that used the GitLab CLI to generate multiple solution branches and run verification builds in parallel. This strategy replaced hours of manual reverse engineering with minutes of automated analysis, allowing the team to debug and validate aggressive optimizations at scale.
Nguyen Van MinhGrab ·
Grab's Mac Cloud Exit supercharges macOS CI/CD
Scaling iOS CI/CD workloads across multiple large mobile applications led to substantial expenses and queue delays under rented cloud macOS infrastructure. To mitigate network latency to Git servers and cut compute overhead, Grab relocated its build fleet from a US cloud provider to a colocation data center in Malaysia. The infrastructure deployment relies on over 200 bare-metal Mac minis across 42RU racks, purposefully avoiding virtualization solutions that introduced performance and stability penalties. Phased migration and guardrail monitoring verified cluster stability while achieving 20% to 40% reductions in pipeline and app build times. Operating the regional bare-metal cluster over an estimated three-year hardware replacement lifecycle is projected to yield 2.4 million USD in total cost of ownership savings.
Madushan GamageGrab ·
Supercharging LLM application development with LLM-Kit
Grab developed LLM-Kit to resolve standardization, security, observability, and infrastructure provisioning bottlenecks across its generative AI development efforts. Submitting an initial application request automatically triggers project generation in GitLab, outputting a modular codebase along with Terraform definitions for Amazon EKS and ECR. The scaffolded architecture incorporates FastAPI, LangChain, OpenID Connect authentication helpers, and PGVector alongside HashiCorp Vault for credential management. Observability and quality tracking are built in through direct integrations with Datadog and LangSmith evaluations. By providing these standardized paved-road templates, the framework has onboarded hundreds of generative AI applications and saved teams an estimated 1.5 weeks of initial setup time.
Boon Zhan ChewGrab ·
Profile-guided optimisation (PGO) on Grab services
Profile-guided optimisation (PGO) improves Go application performance by feeding production CPU profile data into subsequent compiler builds. Teams evaluated PGO across multiple Grab services, including TalariaDB and an orchestrator within the Catwalk service, using Go 1.20 and Go 1.21 builds. TalariaDB saw substantial resource savings after compiling Docker images with a 360-second pprof profile, resulting in at least a 10% drop in CPU usage, a 30% drop in memory consumption, and a 38% reduction in persistent volume usage for event ingestion queues. Conversely, testing on Catwalk showed only a 5% performance gain after adjusting the profiling duration from 59 seconds to 6 minutes, an outcome deemed insufficient to justify pipeline modifications. Broader adoption across monorepo services remains constrained until the central build process supports PGO arguments and automated profile retrieval.
Yonghao HuGrab ·
No version left behind: Our epic journey of GitLab upgrades
Grab's self-hosted GitLab instance fell approximately 14 months behind official releases, serving thousands of engineers and monorepos reaching up to 39TB in total footprint. To address security updates and system stability, the team established a structured upgrade routine using Terraform, Packer, and Ansible across GitLab's 5,000-user reference architecture. They implemented staged deployments, upgraded stateful Gitaly nodes via in-place rotation, and resolved primary node clustering bottlenecks under Praefect. By utilizing official GitLab Upgrade Paths and parallelizing component deployments, the team caught up on 24 months of releases in 11 months and reduced release lag from 396 days down to 35 days.
Saurabh VajpayeeGrab ·
Android App Size at Scale with Project Bonsai
Grab launched Project Bonsai to optimize the size of its Android superapp, which encompassed over four million lines of code across 1,500 modules. Because large app sizes negatively impact user acquisition on low-end devices with constrained bandwidth, the engineering team structured its optimization strategy around measurement, reduction, and containment. The team developed App Sizer, a custom CI-integrated tool that tracks binary compositions and module size contributions in Grafana. Initial reductions targeted Java and Kotlin dex files, resolving bloated transitive R classes by upgrading the Android Gradle Plugin to eliminate overly broad R8 retention rules. These initiatives achieved a 26% reduction in app download size while simultaneously decreasing overall disk footprint.
Nguyen Van MinhGrab ·
An elegant platform
Grab’s Coban real-time data streaming team initially relied on direct Terraform merge requests for managing self-served resources like Kafka topics and CDC pipelines. This pure Infrastructure-as-Code workflow faced challenges with CI pipeline failures from manual errors, security risks from lacking access controls, and platform team review bottlenecks. To resolve this, Coban implemented a three-tier control plane comprising the Coban UI, a Go backend named Heimdall, and a Git storage and provisioner repository named Khone. Heimdall translates user form submissions into validated merge requests containing Terraform and metadata files, polling Khone's CI status and alerting users. This architecture abstracts infrastructure code behind a graphical self-service portal while preserving underlying Git auditing and automation.
Fabrice HarbulotGrab ·
Go module proxy at Grab
Grab's 69.3 GiB multi-module Go monorepo caused commands like go get to take over 18 minutes as Git repeatedly traversed commit history, downloaded large worktrees, and overloaded their GitLab VCS infrastructure. To bypass direct VCS queries without losing automatic updates for external repositories, the team deployed the Athens Go module proxy configured in fallback network mode. They used the GOVCS environment variable to disable Git access specifically for the monorepo path, forcing Athens to fall back to its internal object storage when resolving monorepo modules. A dedicated CI pipeline pre-populates and refreshes the Athens cache whenever new monorepo modules are released. This setup reduced monorepo go get execution times to approximately 12 seconds and allowed a 70% scale-down of the Athens proxy cluster.
Jerry NgGrab ·
How we improved our iOS CI infrastructure with observability tools
Upgrading to Xcode 13.1 introduced severe CI test instability and high CPU utilisation for Grab's iOS development team. To address this, the team integrated observability tools across their UITest pipeline to pinpoint performance bottlenecks and test flakiness. Interventions included isolating spotlight.app to curb CPU spikes, replacing Safari with a mock browser for deep link tests, and booting simulators with pre-granted permissions. The team also built custom network tracking tools to enforce resource mocking and transitioned tests away from arbitrary sleep commands to explicit wait wrappers. These combined changes cut CI runtimes, decreased CPU utilisation by over 50%, and stabilized automated test executions.
Bunty MadanGrab ·
Safer deployment of streaming applications
Stateful stream processing frameworks like Apache Flink present unique deployment challenges because conventional canary and blue-green strategies can cause data inaccuracies or state divergence. Grab's real-time data platform team encountered risks of state loss, manual rollback overhead, and absent health checks in their Kubernetes and Spinnaker deployment pipeline. To resolve these operational issues, the team redesigned the deployment workflow around automated Flink savepointing and programmatic health monitoring. The new pipeline halts existing applications after capturing state snapshots and Kafka offsets, monitors target deployments via API health probes, and executes automated rollbacks using versioned ConfigMaps and replica metadata annotations. This automated process ensures state consistency during upgrades and eliminates manual intervention during deployment failures.
Shi Kai Ng