Orchestrating Chaos Using Grab's Experimentation Platform

Summary

Grab operates hundreds of microservices where failures in non-critical components can cause outages in critical user flows if fallback mechanisms are improperly configured. To validate system resilience, Grab built Chaos ExP by layering a chaos engineering SDK and dedicated web UI on top of its existing Experimentation Platform. Integrated directly into the Grab-Kit server middleware, the framework intercepts incoming requests and evaluates whether to inject failures using local variable resolution. Supported failure primitives include latency, errors, panics, rate throttling, and resource leaks to test dependent services. Combining chaos testing with experimentation telemetry enables engineers to correlate injected infrastructure disruptions with business metric impacts.

Context

Hundreds of distributed microservices run at Grab, where unhandled failures in non-critical services during RPC calls could lead to full outages if contingency strategies like retries and circuit breakers are misconfigured.

Approach / What changed

Building a Chaos SDK and dedicated UI on top of Grab's Experimentation Platform (ExP) and embedding it within the Grab-Kit server middleware to inject controlled application-level failure modes during runtime.

Takeaways

• Integrating the Chaos SDK into Grab-Kit middleware enables automatic failure evaluation on incoming requests without requiring engineers to modify individual endpoint handlers.
• The underlying Experimentation Platform SDK resolves failure variables in single-digit microseconds locally without making external network calls.
• Chaos ExP supports eight failure primitives: errors, CPU load, memory leaks, latency, disk space consumption, goroutine leaks, panics, and request throttling.

Related reading

• Grab · Jul 13, 2018

  Building Grab’s Experimentation Platform

  Grab built its internal Experimentation Platform (ExP) to replace a manual, expensive testing process that required bespoke meetings, custom logging pipelines, and service modifications for each experiment. ExP provides a unified infrastructure featuring a centralized management UI, automated real-time data streaming to S3, and SDKs for Android, iOS, and Go. The platform leverages JSON-based experiment definitions delivered through dynamic configuration management, enabling client-side evaluation without costly network calls. It addresses marketplace network effects and inter-experiment interference through mechanisms such as geo-temporal segmentation and domain-layer models. The platform has scaled to run approximately 25 concurrent experiments while computing roughly 2,500 metrics and 50,000 experiment-metric combinations daily.

  Abeesh Thomas

• Grab · Mar 27, 2019

  Designing Resilient Systems Beyond Retries (Part 3): Architecture Patterns and Chaos Engineering

  Building resilient systems requires architectural safeguards and proactive testing beyond basic retries and circuit breakers. Architectural patterns such as idempotency keys enable safe retries without creating inconsistent state during failures. Asynchronous responses and deferrable work isolate services from downstream dependency latency and errors, though they can conflict with the fail-fast principle. To validate system behavior under stress, chaos engineering introduces intentional failures in production to test hypotheses against a defined steady state. Selectively adopting complementary patterns reduces failure points while avoiding unnecessary architectural complexity.

  Michael Cartmell

• Grab · Apr 9, 2024

  Grab Experiment Decision Engine - a Unified Toolkit for Experimentation

  Grab's Experimentation platform tests thousands of experimental variants weekly, but analyses were largely handled manually using disparate tools unsuited to multi-sided marketplace nuances. To standardize workflows and eliminate reliability issues, Grab developed the GrabX Decision Engine, an internally open-sourced Python toolkit integrated with its Automated Experiment Analysis application. The framework includes a Trusted Advisor component that automates power analysis and sample size estimation while adjusting for clustered standard errors across metric data types. It also bundles a post-experiment statistical toolbox executing default and custom evaluations, ranging from non-parametric tests to regression models that control for confounders and fixed effects. This unified platform institutionalizes experimentation best practices across teams, streamlining collaboration and ensuring consistent statistical inference.

  Ruike Zhang

• Netflix · Jun 19, 2026

  The Data Canary: How Netflix Validates Catalog Metadata

  A manual mitigation action during an incident corrupted a data feed for a subset of titles, causing playback issues and catalog service failures that existing code canary systems failed to catch. To protect streaming reliability, Netflix built an automated data canary system that validates transformed catalog metadata prior to publication. The architecture utilizes a dedicated orchestrator alongside permanent baseline and canary service clusters to coordinate validation using real production traffic. By leveraging custom chaos experiment thresholds, sticky session affinity, and Starts Per Second playback metrics, the system detects regressions in under ten minutes and blocks publication automatically. Controlled failure injection experiments routing approximately 0.2% of global traffic confirmed that issues could be identified in 2.5 to 4 minutes.

  Netflix Technology Blog