Go module proxy at Grab

Summary

Grab's 69.3 GiB multi-module Go monorepo caused commands like go get to take over 18 minutes as Git repeatedly traversed commit history, downloaded large worktrees, and overloaded their GitLab VCS infrastructure. To bypass direct VCS queries without losing automatic updates for external repositories, the team deployed the Athens Go module proxy configured in fallback network mode. They used the GOVCS environment variable to disable Git access specifically for the monorepo path, forcing Athens to fall back to its internal object storage when resolving monorepo modules. A dedicated CI pipeline pre-populates and refreshes the Athens cache whenever new monorepo modules are released. This setup reduced monorepo go get execution times to approximately 12 seconds and allowed a 70% scale-down of the Athens proxy cluster.

Context

Executing Go commands like go get and go list against Grab's massive 69.3 GiB Go monorepo took up to 18 minutes, severely slowing developer workflows and overloading GitLab VCS servers with heavy Git operations.

Approach / What changed

Deploying the Athens Go module proxy using fallback network mode alongside GOVCS settings that disable VCS lookups exclusively for the monorepo, backed by a custom CI cache-refresh pipeline on module releases.

Takeaways

• Running go get on a multi-module repository with 64.7k directories and 10.7k refs triggered expensive Git tree traversals that took over 18 minutes with a local cache and over an hour without one.
• Setting GOVCS to disable VCS access for the monorepo while configuring Athens in fallback mode directs monorepo module queries to storage while preserving automatic VCS lookups for other repositories.
• Bypassing direct VCS hits during Go module retrieval reduced go get duration to ~12 seconds and decreased Athens cluster CPU and memory usage enough to scale the cluster down by 70%.

Related reading

• Grab · Feb 21, 2023

  New zoom freezing feature for Geohash plugin

  Grab introduced a zoom freeze capability to its Geohash plugin for the Java OpenStreetMap Editor (JOSM). Previously, zooming in or out forced the plugin to calculate and render new geohashes automatically, causing visual clutter when users needed to focus on a specific region. The updated interface introduces a toggle label allowing editors to switch between enabling and disabling zoom freezing. While frozen, the current geohash boundaries remain locked during map scaling, and options to show smaller or larger geohashes are disabled. Disabling the feature instantly updates the display with the appropriate geohash subdivisions for the current zoom level.

  Maria Mitisor

• Grab · Apr 9, 2024

  Grab Experiment Decision Engine - a Unified Toolkit for Experimentation

  Grab's Experimentation platform tests thousands of experimental variants weekly, but analyses were largely handled manually using disparate tools unsuited to multi-sided marketplace nuances. To standardize workflows and eliminate reliability issues, Grab developed the GrabX Decision Engine, an internally open-sourced Python toolkit integrated with its Automated Experiment Analysis application. The framework includes a Trusted Advisor component that automates power analysis and sample size estimation while adjusting for clustered standard errors across metric data types. It also bundles a post-experiment statistical toolbox executing default and custom evaluations, ranging from non-parametric tests to regression models that control for confounders and fixed effects. This unified platform institutionalizes experimentation best practices across teams, streamlining collaboration and ensuring consistent statistical inference.

  Ruike Zhang

• Grab · Jun 2, 2022

  Graph concepts and applications

  Real-world systems generate dynamic, non-random connections that traditional statistical approaches fail to characterize or forecast. Graph models represent these structures through vertices and edges, abstracting complex networks into mathematically tractable relationships. Common data representation formats include the Resource Description Framework (RDF), which models subject-predicate-object triples with IRIs, literals, and blank nodes, and Labeled Property Graphs (LPGs), which store arbitrary key-value properties directly on nodes and edges. Graph databases, derived from the LPG model, treat relationships with equal weight to entities, delivering responsive traversals for highly interconnected systems. While they offer agility and explicit relationship modeling, graph databases lack a standardized query language and remain poorly suited for standard transaction-focused workloads compared to relational databases.

  Wenxiang Lu

• Grab · May 23, 2024

  How we evaluated the business impact of marketing campaigns

  Grab needed a scalable, centralised way to evaluate the business impact of millions of marketing messages sent across expanding campaigns. Manual analysis by marketing analysts produced inconsistent results and failed to scale, leading to the creation of multi-touch attribution and controlled experiment methodologies. The team designed a Marketing attribution model (MAM) that credits touchpoints based on relevance and recency, while using stratified hold-out groups to assess single campaign lift against baseline metrics. Statistical significance calculations, implemented using the jStat library based on the central limit theorem, ensure observed business metric differences like gross merchandise value are not random. The system provides campaign managers with direct access to impact analyses while the team works on challenges surrounding smaller sample sizes and hold-out group constraints.

  Jie Zhang