Automated Experiment Analysis - Making experimental analysis scalable

Summary

Manual ad-hoc analysis of online controlled experiments at Grab introduced operational inefficiencies, inconsistent quality control, and scalability barriers across teams. To resolve these issues, Grab extended its GrabX experimentation platform with an Automated Experiment Analysis system that standardises metrics and automates statistical evaluations. The architecture stores experiment configurations and metric definitions from Cosmos DB into Azure Data Lake as bronze datasets, uses Spark on Databricks via Azure Data Factory to process subjects into silver datasets, and applies an internal Python Decision Engine to generate final gold results. These gold datasets are stored in star-schema fact and dimension tables and presented directly in the GrabX interface using embedded Power BI visualisations. The automation eliminates repetitive data pipeline construction for analysts, ensures reproducible findings aligned with initial hypotheses, and accelerates product launch decisions.

Takeaways

1. 1

   The pipeline separates data processing into bronze metadata datasets, silver metrics computation datasets, and gold star-schema result datasets.

2. 2

   Decoupling statistical testing logic into an internal Python package named Decision Engine enables crowdsourced contributions and modular additions of advanced statistical techniques.

3. 3

   Users can configure metrics using a glossary API with pre-defined metrics from the Scribe datamart, specify funnel event-based metrics, or supply custom SQL queries.