Reliable and Scalable Feature Toggles and A/B Testing SDK at Grab

Summary

Grab previously managed experiments using custom service-level code and a toggling library that queried a shared Redis instance, creating latency risks and a single point of failure across backend microservices. To achieve reliable, sub-microsecond feature evaluations, the team designed a Go SDK that resolves rollouts and A/B tests entirely in memory without runtime network I/O. Backend services periodically poll JSON-defined configuration schemas stored in Amazon S3 through a Universal Configuration Manager. The SDK evaluates contextual attributes called facets locally and pushes decision telemetry asynchronously to an S3 and Presto data lake. This architecture allows engineering and product teams to gate deployments and run server-side experiments safely without service disruption.

Context

Manual experimentation code and a legacy feature toggling library relying on network calls to a shared Redis created operational friction, latency risks, and a single point of failure as Grab's microservices expanded.

Approach / What changed

Building an intelligent Go SDK that loads JSON rollout and experiment definitions from Amazon S3 into local memory for sub-microsecond evaluations while logging decision metrics asynchronously.

Takeaways

• Eliminating runtime network I/O by executing feature toggle and experiment logic in client memory prevents network jitter and service-to-service bottlenecks.
• Standardized context attributes called facets allow the SDK to evaluate granular targeting constraints—such as city, driver, or passenger IDs—locally on each request.
• Decision tracking is decoupled from variable retrieval by queueing and reporting evaluation metadata asynchronously to the analytics data pipeline.

Related reading

• Grab · Jul 13, 2018

  Building Grab’s Experimentation Platform

  Grab built its internal Experimentation Platform (ExP) to replace a manual, expensive testing process that required bespoke meetings, custom logging pipelines, and service modifications for each experiment. ExP provides a unified infrastructure featuring a centralized management UI, automated real-time data streaming to S3, and SDKs for Android, iOS, and Go. The platform leverages JSON-based experiment definitions delivered through dynamic configuration management, enabling client-side evaluation without costly network calls. It addresses marketplace network effects and inter-experiment interference through mechanisms such as geo-temporal segmentation and domain-layer models. The platform has scaled to run approximately 25 concurrent experiments while computing roughly 2,500 metrics and 50,000 experiment-metric combinations daily.

  Abeesh Thomas

• Grab · Apr 9, 2024

  Grab Experiment Decision Engine - a Unified Toolkit for Experimentation

  Grab's Experimentation platform tests thousands of experimental variants weekly, but analyses were largely handled manually using disparate tools unsuited to multi-sided marketplace nuances. To standardize workflows and eliminate reliability issues, Grab developed the GrabX Decision Engine, an internally open-sourced Python toolkit integrated with its Automated Experiment Analysis application. The framework includes a Trusted Advisor component that automates power analysis and sample size estimation while adjusting for clustered standard errors across metric data types. It also bundles a post-experiment statistical toolbox executing default and custom evaluations, ranging from non-parametric tests to regression models that control for confounders and fixed effects. This unified platform institutionalizes experimentation best practices across teams, streamlining collaboration and ensuring consistent statistical inference.

  Ruike Zhang

• Grab · May 30, 2022

  Automated Experiment Analysis - Making experimental analysis scalable

  Manual ad-hoc analysis of online controlled experiments at Grab introduced operational inefficiencies, inconsistent quality control, and scalability barriers across teams. To resolve these issues, Grab extended its GrabX experimentation platform with an Automated Experiment Analysis system that standardises metrics and automates statistical evaluations. The architecture stores experiment configurations and metric definitions from Cosmos DB into Azure Data Lake as bronze datasets, uses Spark on Databricks via Azure Data Factory to process subjects into silver datasets, and applies an internal Python Decision Engine to generate final gold results. These gold datasets are stored in star-schema fact and dimension tables and presented directly in the GrabX interface using embedded Power BI visualisations. The automation eliminates repetitive data pipeline construction for analysts, ensures reproducible findings aligned with initial hypotheses, and accelerates product launch decisions.

  Albert Cheng

• Grab · Jan 16, 2019

  A Lean and Scalable Data Pipeline to Capture Large Scale Events and Support Experimentation Platform

  Controlled online experimentation across diverse product verticals requires tracking interactions across systems to prevent local optimizations from causing global degradation. Grab built a batch data pipeline to capture, ingest, and process petabytes of event data to support its experimentation platform and analytics stakeholders. The architecture loads ingested event data from Amazon S3, transforms and sorts it, and writes partitioned output back to S3 with metadata registered in Apache Hive. Using Apache Spark on AWS Elastic MapReduce with Apache Airflow for orchestration, the system handles roughly 400,000 incoming events per second. The data is partitioned by event type and ingestion time and stored in Apache ORC format to streamline query workloads and reduce retrieval overhead.

  Oscar Cassetti