# Automated Experiment Analysis - Making experimental analysis scalable

[Grab](https://yomu.fyi/company/grab) · Albert Cheng · May 30, 2022

**Type:** Problem & solution

## Summary

Manual ad-hoc analysis of online controlled experiments at Grab introduced operational inefficiencies, inconsistent quality control, and scalability barriers across teams. To resolve these issues, Grab extended its GrabX experimentation platform with an Automated Experiment Analysis system that standardises metrics and automates statistical evaluations. The architecture stores experiment configurations and metric definitions from Cosmos DB into Azure Data Lake as bronze datasets, uses Spark on Databricks via Azure Data Factory to process subjects into silver datasets, and applies an internal Python Decision Engine to generate final gold results. These gold datasets are stored in star-schema fact and dimension tables and presented directly in the GrabX interface using embedded Power BI visualisations. The automation eliminates repetitive data pipeline construction for analysts, ensures reproducible findings aligned with initial hypotheses, and accelerates product launch decisions.

## Context

Ad-hoc manual analysis of online controlled experiments at Grab caused operational inefficiencies, risk of unstandardised and inaccurate results, and poor knowledge sharing across teams with varied experimentation skill sets.

## Approach / What changed

Grab developed Automated Experiment Analysis, an automated pipeline integrated with GrabX that stores metrics configurations in Cosmos DB, transforms data through bronze, silver, and gold layers using Azure Data Factory and Databricks Spark jobs, runs statistical tests via a decoupled internal Python package called Decision Engine, and displays results in GrabX using embedded Power BI visualisations.

## Takeaways

- The pipeline separates data processing into bronze metadata datasets, silver metrics computation datasets, and gold star-schema result datasets.
- Decoupling statistical testing logic into an internal Python package named Decision Engine enables crowdsourced contributions and modular additions of advanced statistical techniques.
- Users can configure metrics using a glossary API with pre-defined metrics from the Scribe datamart, specify funnel event-based metrics, or supply custom SQL queries.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Azure](https://yomu.fyi/topic/azure), [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Python](https://yomu.fyi/topic/python), [Scalability](https://yomu.fyi/topic/scalability)

[Read original post](https://engineering.grab.com/automated-experiment-analysis)