# Reliable and Scalable Feature Toggles and A/B Testing SDK at Grab

[Grab](https://yomu.fyi/company/grab) · Roman Atachiants · Nov 2, 2018

**Type:** Problem & solution

## Summary

Grab previously managed experiments using custom service-level code and a toggling library that queried a shared Redis instance, creating latency risks and a single point of failure across backend microservices. To achieve reliable, sub-microsecond feature evaluations, the team designed a Go SDK that resolves rollouts and A/B tests entirely in memory without runtime network I/O. Backend services periodically poll JSON-defined configuration schemas stored in Amazon S3 through a Universal Configuration Manager. The SDK evaluates contextual attributes called facets locally and pushes decision telemetry asynchronously to an S3 and Presto data lake. This architecture allows engineering and product teams to gate deployments and run server-side experiments safely without service disruption.

## Context

Manual experimentation code and a legacy feature toggling library relying on network calls to a shared Redis created operational friction, latency risks, and a single point of failure as Grab's microservices expanded.

## Approach / What changed

Building an intelligent Go SDK that loads JSON rollout and experiment definitions from Amazon S3 into local memory for sub-microsecond evaluations while logging decision metrics asynchronously.

## Takeaways

- Eliminating runtime network I/O by executing feature toggle and experiment logic in client memory prevents network jitter and service-to-service bottlenecks.
- Standardized context attributes called facets allow the SDK to evaluate granular targeting constraints—such as city, driver, or passenger IDs—locally on each request.
- Decision tracking is decoupled from variable retrieval by queueing and reporting evaluation metadata asynchronously to the analytics data pipeline.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Go](https://yomu.fyi/topic/go), [Reliability](https://yomu.fyi/topic/reliability), [Scalability](https://yomu.fyi/topic/scalability), [Testing](https://yomu.fyi/topic/testing)

[Read original post](https://engineering.grab.com/feature-toggles-ab-testing)