How we reduced our CI YAML files from 1800 lines to 50 lines

Summary

Grab's Cauldron Machine Learning Platform team managed continuous delivery across multiple pipelines by using nested GitLab CI configurations, but soon encountered platform limitations such as the 100-include ceiling and bloated 1,800-line YAML files. To address these constraints, the team implemented GitLab Dynamic Child Pipelines to programmatically create execution stages on the fly. They built a command-line utility in Rust that runs git diff against the base branch, extracts pipeline and stage metadata using configurable stop patterns, and applies a template to produce a dynamic CI definition. On the master branch, the tool fetches the diff artifact from the source branch through the GitLab API to execute identical generation logic. This architectural shift reduced the root CI file from 1,800 lines down to a constant 50 lines, allowing configuration size to remain stable regardless of repository growth.

Takeaways

1. 1

   GitLab imposes a default soft limit of 100 include statements in base `.gitlab-ci.yml` files, alongside file size constraints.

2. 2

   A custom Rust tool parsed `git diff` paths into template variables using stop patterns like `pipeline_name:pipelines` to dynamically generate stage configs.

3. 3

   Dynamic Child Pipelines reduced the repository's root CI configuration from 1,800 lines to a fixed 50 lines that scales independently of added pipelines.