Automated Experiment Analysis - Making experimental analysis scalable

Summary

Manual ad-hoc analysis of online controlled experiments at Grab introduced operational inefficiencies, inconsistent quality control, and scalability barriers across teams. To resolve these issues, Grab extended its GrabX experimentation platform with an Automated Experiment Analysis system that standardises metrics and automates statistical evaluations. The architecture stores experiment configurations and metric definitions from Cosmos DB into Azure Data Lake as bronze datasets, uses Spark on Databricks via Azure Data Factory to process subjects into silver datasets, and applies an internal Python Decision Engine to generate final gold results. These gold datasets are stored in star-schema fact and dimension tables and presented directly in the GrabX interface using embedded Power BI visualisations. The automation eliminates repetitive data pipeline construction for analysts, ensures reproducible findings aligned with initial hypotheses, and accelerates product launch decisions.

Context

Ad-hoc manual analysis of online controlled experiments at Grab caused operational inefficiencies, risk of unstandardised and inaccurate results, and poor knowledge sharing across teams with varied experimentation skill sets.

Approach / What changed

Grab developed Automated Experiment Analysis, an automated pipeline integrated with GrabX that stores metrics configurations in Cosmos DB, transforms data through bronze, silver, and gold layers using Azure Data Factory and Databricks Spark jobs, runs statistical tests via a decoupled internal Python package called Decision Engine, and displays results in GrabX using embedded Power BI visualisations.

Takeaways

• The pipeline separates data processing into bronze metadata datasets, silver metrics computation datasets, and gold star-schema result datasets.
• Decoupling statistical testing logic into an internal Python package named Decision Engine enables crowdsourced contributions and modular additions of advanced statistical techniques.
• Users can configure metrics using a glossary API with pre-defined metrics from the Scribe datamart, specify funnel event-based metrics, or supply custom SQL queries.

Related reading

• Grab · Nov 16, 2022

  Query expansion based on user behaviour

  Grab developed a query expansion framework to resolve search issues such as typos, multi-language terms, missing alternatives, and word spacing. The corpus is generated through manual annotation for the top thousand fat-head queries and session-based data mining of user rewrite patterns. Offline pipelines extract adjacent query rewrites occurring within 30 seconds where only the second query generated a click, limiting candidates to three per query for serving efficiency. In production, expanded keywords pass through a query understanding, multi-recall, and result fusion flow with automated pruning of underperforming pairs. Online A/B testing across six Southeast Asian countries showed improvements in click-through and conversion rates, with data-mined expansions outperforming manual annotations alone.

  Shuailong Liang

• Grab · Aug 2, 2023

  Unsupervised graph anomaly detection - Catching new fraudulent behaviours

  Fraud detection systems relying on historical training data often struggle to detect new fraudulent modus operandi because emerging patterns lack labeled supervision. Grab developed GraphBEAN, an unsupervised autoencoder model designed to detect anomalous patterns across bipartite interaction graphs representing consumers and merchants. The architecture uses graph convolution layers to encode node and edge attributes, then reconstructs features and edge existence through separate feature and structure decoders. High reconstruction errors flag rare, anomalous behaviors at both the node and edge levels. An automated pipeline applies heuristic fraud tags to these scores and routes them to human analysts and automated mitigation systems.

  Rizal Fathony

• Grab · Oct 25, 2022

  Using mobile sensor data to encourage safer driving

  Dedicated vehicular inertial measurement units are prohibitively expensive to install across large fleets, while customer feedback lacks the coverage required to assess driving safety comprehensively. To monitor driver behavior at scale, Grab processes smartphone sensor data collected via a mobile telematics SDK at high sampling frequencies. A custom orientation algorithm calculates Euler roll and pitch angles from gravity and resolves yaw through principal component analysis to align phone measurements with vehicle axes. The pipeline applies low-pass or Kalman filtering to handle varying noise levels, followed by threshold peak-finding algorithms to detect harsh acceleration, braking, and cornering. Speeding events are identified by comparing GPS speeds against known map limits or aggregated speed profiles derived from free-flowing traffic.

  Laiyi Lin

• Grab · Nov 8, 2023

  Graph modelling guidelines

  Graph modelling uses graph theory to represent real-world entities, relationships, and properties through nodes and edges, unlocking insights across interconnected datasets. The process begins with defining domain concepts, mapping entities and relationships, and assigning relevant descriptive properties to both nodes and edges. Practitioners choose between graph structures such as property graphs or Resource Description Framework models before optionally developing a structural schema. Data is then imported or generated to populate storage systems such as Neo4j, Amazon Neptune, Azure Cosmos DB, or in-memory data structures. Finally, the graph is visualised and analysed using algorithmic methods like shortest path calculation, centrality measurement, and community detection.

  Wenxiang Lu