Grab Experiment Decision Engine - a Unified Toolkit for Experimentation

Summary

Grab's Experimentation platform tests thousands of experimental variants weekly, but analyses were largely handled manually using disparate tools unsuited to multi-sided marketplace nuances. To standardize workflows and eliminate reliability issues, Grab developed the GrabX Decision Engine, an internally open-sourced Python toolkit integrated with its Automated Experiment Analysis application. The framework includes a Trusted Advisor component that automates power analysis and sample size estimation while adjusting for clustered standard errors across metric data types. It also bundles a post-experiment statistical toolbox executing default and custom evaluations, ranging from non-parametric tests to regression models that control for confounders and fixed effects. This unified platform institutionalizes experimentation best practices across teams, streamlining collaboration and ensuring consistent statistical inference.

Context

Manual and federated experiment analyses led to a lack of marketplace-specific design tools, inconsistent statistical standards, and inefficient, non-scalable evaluation workflows across Grab's tech teams.

Approach / What changed

Grab created the GrabX Decision Engine, an internal open-source Python package that integrates a pre-experiment Trusted Advisor for power and sample size calculations with a comprehensive post-experiment statistical analysis toolbox.

Takeaways

• Between-subject designs with multiple observations per participant create high within-subject correlation, requiring adjustments for clustered standard errors during power analysis and hypothesis testing.
• Switchback experiments in multi-sided marketplaces treat timeslices as experimental units, introducing challenges like treatment-window autocorrelation, cross-slice spillovers, and sample ratio mismatch (SRM).
• The Decision Engine's Trusted Advisor accepts PySpark or Pandas dataframes and automatically adapts power analysis calculations based on detected properties like binary decision variables.

Related reading

• Grab · Jun 30, 2023

  Go module proxy at Grab

  Grab's 69.3 GiB multi-module Go monorepo caused commands like go get to take over 18 minutes as Git repeatedly traversed commit history, downloaded large worktrees, and overloaded their GitLab VCS infrastructure. To bypass direct VCS queries without losing automatic updates for external repositories, the team deployed the Athens Go module proxy configured in fallback network mode. They used the GOVCS environment variable to disable Git access specifically for the monorepo path, forcing Athens to fall back to its internal object storage when resolving monorepo modules. A dedicated CI pipeline pre-populates and refreshes the Athens cache whenever new monorepo modules are released. This setup reduced monorepo go get execution times to approximately 12 seconds and allowed a 70% scale-down of the Athens proxy cluster.

  Jerry Ng

• Grab · Feb 21, 2023

  New zoom freezing feature for Geohash plugin

  Grab introduced a zoom freeze capability to its Geohash plugin for the Java OpenStreetMap Editor (JOSM). Previously, zooming in or out forced the plugin to calculate and render new geohashes automatically, causing visual clutter when users needed to focus on a specific region. The updated interface introduces a toggle label allowing editors to switch between enabling and disabling zoom freezing. While frozen, the current geohash boundaries remain locked during map scaling, and options to show smaller or larger geohashes are disabled. Disabling the feature instantly updates the display with the appropriate geohash subdivisions for the current zoom level.

  Maria Mitisor

• Grab · Jun 2, 2022

  Graph concepts and applications

  Real-world systems generate dynamic, non-random connections that traditional statistical approaches fail to characterize or forecast. Graph models represent these structures through vertices and edges, abstracting complex networks into mathematically tractable relationships. Common data representation formats include the Resource Description Framework (RDF), which models subject-predicate-object triples with IRIs, literals, and blank nodes, and Labeled Property Graphs (LPGs), which store arbitrary key-value properties directly on nodes and edges. Graph databases, derived from the LPG model, treat relationships with equal weight to entities, delivering responsive traversals for highly interconnected systems. While they offer agility and explicit relationship modeling, graph databases lack a standardized query language and remain poorly suited for standard transaction-focused workloads compared to relational databases.

  Wenxiang Lu

• Grab · May 23, 2024

  How we evaluated the business impact of marketing campaigns

  Grab needed a scalable, centralised way to evaluate the business impact of millions of marketing messages sent across expanding campaigns. Manual analysis by marketing analysts produced inconsistent results and failed to scale, leading to the creation of multi-touch attribution and controlled experiment methodologies. The team designed a Marketing attribution model (MAM) that credits touchpoints based on relevance and recency, while using stratified hold-out groups to assess single campaign lift against baseline metrics. Statistical significance calculations, implemented using the jStat library based on the central limit theorem, ensure observed business metric differences like gross merchandise value are not random. The system provides campaign managers with direct access to impact analyses while the team works on challenges surrounding smaller sample sizes and hold-out group constraints.

  Jie Zhang