Grab Experiment Decision Engine - a Unified Toolkit for Experimentation

Summary

Grab's Experimentation platform tests thousands of experimental variants weekly, but analyses were largely handled manually using disparate tools unsuited to multi-sided marketplace nuances. To standardize workflows and eliminate reliability issues, Grab developed the GrabX Decision Engine, an internally open-sourced Python toolkit integrated with its Automated Experiment Analysis application. The framework includes a Trusted Advisor component that automates power analysis and sample size estimation while adjusting for clustered standard errors across metric data types. It also bundles a post-experiment statistical toolbox executing default and custom evaluations, ranging from non-parametric tests to regression models that control for confounders and fixed effects. This unified platform institutionalizes experimentation best practices across teams, streamlining collaboration and ensuring consistent statistical inference.

Takeaways

1. 1

   Between-subject designs with multiple observations per participant create high within-subject correlation, requiring adjustments for clustered standard errors during power analysis and hypothesis testing.

2. 2

   Switchback experiments in multi-sided marketplaces treat timeslices as experimental units, introducing challenges like treatment-window autocorrelation, cross-slice spillovers, and sample ratio mismatch (SRM).

3. 3

   The Decision Engine's Trusted Advisor accepts PySpark or Pandas dataframes and automatically adapts power analysis calculations based on detected properties like binary decision variables.