Securing GitOps pipelines

Summary

Grab's real-time data platform team transitioned from an Atlantis-driven Terraform workflow to an in-house GitOps platform called Khone to manage streaming infrastructure resources like Kafka topics and Flink pipelines. The earlier setup suffered from coarse-grained access controls, required manual merge request comments, and lacked flexible validation capabilities within native configuration files. Khone derives environment parameters directly from standardized directory paths and uses Python with the python-hcl2 library to inspect and validate resource definitions before executing Terraform stages in parallel. To prevent configuration tampering in merge requests, CI/CD pipeline definitions and execution scripts are isolated in a separate administrative repository and fetched during job runs using shallow Git clones.

Context

Grab's data platform team experienced limitations with their Atlantis-based Terraform automation, including coarse-grained access controls that allowed unintended resource creation, manual merge request comment requirements, restricted HCL validation capabilities, and boilerplate code across projects.

Approach / What changed

The team built an in-house GitOps platform named Khone that derives configurations from directory paths, validates HCL definitions using Python scripts and python-hcl2, executes parallel Terraform plan and apply jobs, and stores pipeline configurations and execution scripts in a separate private repository.

Takeaways

• Khone derives parameters like Kafka broker endpoints and validates inputs by encoding properties such as namespace, resource type, environment, and cluster name directly into directory paths.
• Pipeline tampering is prevented by isolating GitLab CI configuration files and validation scripts in a separate administrative repository and shallow-cloning them during job execution.
• HCL source code is inspected using the python-hcl2 library to validate module sources, enforce resource whitelists, and restrict resource types without relying on native HCL functions.

Related reading

• Grab · Feb 20, 2024

  The journey of building a comprehensive attribution platform

  Grab needed to modernize its marketing analytics from manual ad hoc queries and high data latency to a platform supporting real-time attribution for pricing models like cost per order. The engineering team initially deployed a pure stream-processing engine using Kappa architecture, Kafka, ScyllaDB, and Redis, which reduced latency from days to minutes and merged ads and promo touchpoints. However, stream-only processing faced high costs, out-of-order event issues, and difficulties running multi-touch models across longer historical windows. Grab transitioned to a Lambda architecture pairing Coban stream processing with Spark-based batch ETL and Amazon S3. This hybrid design separated real-time operational metrics from historical batch reporting, cutting real-time processing costs by approximately 25% while maintaining under 1% data discrepancy.

  Kang Huang

• Grab · Sep 23, 2024

  Bringing Grab’s Live Activity to Android: Enhancing user experience through custom notifications

  Grab designed an equivalent to iOS Live Activities for Android to provide real-time order tracking outside the app. Because Android lacks Apple's native ActivityKit push token system, the team substituted push tokens with placeholder values to maintain technical consistency across platforms and preserve backend push targeting via their Hedwig service. For the user interface, engineering selected custom notifications over floating views because custom notifications avoid intrusive screen usage and do not require the 'Draw over other apps' permission. The client implementation separates responsibilities across dedicated classes: LiveActivityIntegrationManager handles token registration across business verticals, LiveActivityAttributes encapsulates UI configuration, and LiveActivityManager maps payloads to Android NotificationManager instances. The solution launched for Food, Mart, Express, and Transport verticals.

  Jessica Sean

• Grab · Feb 1, 2024

  Managing dynamic marketplace content at scale: Grab's approach to content moderation

  Grab manages content moderation across GrabFood and GrabMart, where merchants update over 100,000 item listings daily across eight countries. The platform must adhere to internal rules, local government regulations, and external platform guidelines across these markets. To handle scale and regional nuance, Grab uses an in-house automated system powered by algorithms and machine learning to scan daily listings for violations. Items flagged as requiring subjective judgment or cultural awareness are routed to human moderators for manual review. Additionally, Grab collaborates with Google to maintain compliance with Play Store policies.

  Poonam Gambhire

• Grab · Sep 22, 2023

  Stepping up marketing for advertisers: Scalable lookalike audience

  Grab's legacy lookalike audience generation platform suffered from long creation SLAs of two working days, high costs, and low weekly update frequencies. To resolve these bottlenecks, the engineering team designed an embedding-based platform powered by an in-memory retrieval service and automated update pipelines. The system creates audience representations by averaging constituent passenger embeddings, determining user membership through real-time cosine score thresholds. To eliminate feature store latency while fitting all embeddings into memory, a hash-based compression method cuts passenger embedding storage needs by roughly 90%. Consequently, audience availability dropped to within 15 minutes of campaign creation, audience generation costs fell by 98%, and ad impressions and clicks doubled.

  William Wu