# How we reduced our CI YAML files from 1800 lines to 50 lines

[Grab](https://yomu.fyi/company/grab) · Jia Long Loh · Apr 19, 2022

**Type:** Problem & solution

## Summary

Grab's Cauldron Machine Learning Platform team managed continuous delivery across multiple pipelines by using nested GitLab CI configurations, but soon encountered platform limitations such as the 100-include ceiling and bloated 1,800-line YAML files. To address these constraints, the team implemented GitLab Dynamic Child Pipelines to programmatically create execution stages on the fly. They built a command-line utility in Rust that runs git diff against the base branch, extracts pipeline and stage metadata using configurable stop patterns, and applies a template to produce a dynamic CI definition. On the master branch, the tool fetches the diff artifact from the source branch through the GitLab API to execute identical generation logic. This architectural shift reduced the root CI file from 1,800 lines down to a constant 50 lines, allowing configuration size to remain stable regardless of repository growth.

## Context

Grab's Cauldron Machine Learning Platform team encountered GitLab CI scalability limitations when maintaining large ML repositories, including a default soft limit of 100 includes and unwieldy configuration files reaching 1,800 lines.

## Approach / What changed

The team adopted GitLab Dynamic Child Pipelines combined with a custom Rust CLI utility that evaluates git diff outputs, uses stop patterns to parse pipeline and stage names from changed file paths, and renders runtime YAML pipeline definitions from templates.

## Takeaways

- GitLab imposes a default soft limit of 100 include statements in base \`.gitlab-ci.yml\` files, alongside file size constraints.
- A custom Rust tool parsed \`git diff\` paths into template variables using stop patterns like \`pipeline\_name:pipelines\` to dynamically generate stage configs.
- Dynamic Child Pipelines reduced the repository's root CI configuration from 1,800 lines to a fixed 50 lines that scales independently of added pipelines.

**Tags:** [CI/CD](https://yomu.fyi/topic/ci-cd), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Rust](https://yomu.fyi/topic/rust)

[Read original post](https://engineering.grab.com/how-we-reduced-our-ci-yaml)