How we reduced our CI YAML files from 1800 lines to 50 lines

Summary

Grab's Cauldron Machine Learning Platform team managed continuous delivery across multiple pipelines by using nested GitLab CI configurations, but soon encountered platform limitations such as the 100-include ceiling and bloated 1,800-line YAML files. To address these constraints, the team implemented GitLab Dynamic Child Pipelines to programmatically create execution stages on the fly. They built a command-line utility in Rust that runs git diff against the base branch, extracts pipeline and stage metadata using configurable stop patterns, and applies a template to produce a dynamic CI definition. On the master branch, the tool fetches the diff artifact from the source branch through the GitLab API to execute identical generation logic. This architectural shift reduced the root CI file from 1,800 lines down to a constant 50 lines, allowing configuration size to remain stable regardless of repository growth.

Context

Grab's Cauldron Machine Learning Platform team encountered GitLab CI scalability limitations when maintaining large ML repositories, including a default soft limit of 100 includes and unwieldy configuration files reaching 1,800 lines.

Approach / What changed

The team adopted GitLab Dynamic Child Pipelines combined with a custom Rust CLI utility that evaluates git diff outputs, uses stop patterns to parse pipeline and stage names from changed file paths, and renders runtime YAML pipeline definitions from templates.

Takeaways

• GitLab imposes a default soft limit of 100 include statements in base `.gitlab-ci.yml` files, alongside file size constraints.
• A custom Rust tool parsed `git diff` paths into template variables using stop patterns like `pipeline_name:pipelines` to dynamically generate stage configs.
• Dynamic Child Pipelines reduced the repository's root CI configuration from 1,800 lines to a fixed 50 lines that scales independently of added pipelines.

Related reading

• Grab · Feb 20, 2024

  The journey of building a comprehensive attribution platform

  Grab needed to modernize its marketing analytics from manual ad hoc queries and high data latency to a platform supporting real-time attribution for pricing models like cost per order. The engineering team initially deployed a pure stream-processing engine using Kappa architecture, Kafka, ScyllaDB, and Redis, which reduced latency from days to minutes and merged ads and promo touchpoints. However, stream-only processing faced high costs, out-of-order event issues, and difficulties running multi-touch models across longer historical windows. Grab transitioned to a Lambda architecture pairing Coban stream processing with Spark-based batch ETL and Amazon S3. This hybrid design separated real-time operational metrics from historical batch reporting, cutting real-time processing costs by approximately 25% while maintaining under 1% data discrepancy.

  Kang Huang

• Grab · Sep 23, 2024

  Bringing Grab’s Live Activity to Android: Enhancing user experience through custom notifications

  Grab designed an equivalent to iOS Live Activities for Android to provide real-time order tracking outside the app. Because Android lacks Apple's native ActivityKit push token system, the team substituted push tokens with placeholder values to maintain technical consistency across platforms and preserve backend push targeting via their Hedwig service. For the user interface, engineering selected custom notifications over floating views because custom notifications avoid intrusive screen usage and do not require the 'Draw over other apps' permission. The client implementation separates responsibilities across dedicated classes: LiveActivityIntegrationManager handles token registration across business verticals, LiveActivityAttributes encapsulates UI configuration, and LiveActivityManager maps payloads to Android NotificationManager instances. The solution launched for Food, Mart, Express, and Transport verticals.

  Jessica Sean

• Grab · Feb 1, 2024

  Managing dynamic marketplace content at scale: Grab's approach to content moderation

  Grab manages content moderation across GrabFood and GrabMart, where merchants update over 100,000 item listings daily across eight countries. The platform must adhere to internal rules, local government regulations, and external platform guidelines across these markets. To handle scale and regional nuance, Grab uses an in-house automated system powered by algorithms and machine learning to scan daily listings for violations. Items flagged as requiring subjective judgment or cultural awareness are routed to human moderators for manual review. Additionally, Grab collaborates with Google to maintain compliance with Play Store policies.

  Poonam Gambhire

• Grab · Sep 22, 2023

  Stepping up marketing for advertisers: Scalable lookalike audience

  Grab's legacy lookalike audience generation platform suffered from long creation SLAs of two working days, high costs, and low weekly update frequencies. To resolve these bottlenecks, the engineering team designed an embedding-based platform powered by an in-memory retrieval service and automated update pipelines. The system creates audience representations by averaging constituent passenger embeddings, determining user membership through real-time cosine score thresholds. To eliminate feature store latency while fitting all embeddings into memory, a hash-based compression method cuts passenger embedding storage needs by roughly 90%. Consequently, audience availability dropped to within 15 minutes of campaign creation, audience generation costs fell by 98%, and ad impressions and clicks doubled.

  William Wu