# Grab Experiment Decision Engine - a Unified Toolkit for Experimentation

[Grab](https://yomu.fyi/company/grab) · Ruike Zhang · Apr 9, 2024

**Type:** Problem & solution

## Summary

Grab's Experimentation platform tests thousands of experimental variants weekly, but analyses were largely handled manually using disparate tools unsuited to multi-sided marketplace nuances. To standardize workflows and eliminate reliability issues, Grab developed the GrabX Decision Engine, an internally open-sourced Python toolkit integrated with its Automated Experiment Analysis application. The framework includes a Trusted Advisor component that automates power analysis and sample size estimation while adjusting for clustered standard errors across metric data types. It also bundles a post-experiment statistical toolbox executing default and custom evaluations, ranging from non-parametric tests to regression models that control for confounders and fixed effects. This unified platform institutionalizes experimentation best practices across teams, streamlining collaboration and ensuring consistent statistical inference.

## Context

Manual and federated experiment analyses led to a lack of marketplace-specific design tools, inconsistent statistical standards, and inefficient, non-scalable evaluation workflows across Grab's tech teams.

## Approach / What changed

Grab created the GrabX Decision Engine, an internal open-source Python package that integrates a pre-experiment Trusted Advisor for power and sample size calculations with a comprehensive post-experiment statistical analysis toolbox.

## Takeaways

- Between-subject designs with multiple observations per participant create high within-subject correlation, requiring adjustments for clustered standard errors during power analysis and hypothesis testing.
- Switchback experiments in multi-sided marketplaces treat timeslices as experimental units, introducing challenges like treatment-window autocorrelation, cross-slice spillovers, and sample ratio mismatch (SRM).
- The Decision Engine's Trusted Advisor accepts PySpark or Pandas dataframes and automatically adapts power analysis calculations based on detected properties like binary decision variables.

**Tags:** [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Open Source](https://yomu.fyi/topic/open-source), [Python](https://yomu.fyi/topic/python), [Testing](https://yomu.fyi/topic/testing)

[Read original post](https://engineering.grab.com/grabx-decision-engine)