# Orchestrating Chaos Using Grab's Experimentation Platform

[Grab](https://yomu.fyi/company/grab) · Roman Atachiants · Nov 23, 2018

**Type:** Problem & solution

## Summary

Grab operates hundreds of microservices where failures in non-critical components can cause outages in critical user flows if fallback mechanisms are improperly configured. To validate system resilience, Grab built Chaos ExP by layering a chaos engineering SDK and dedicated web UI on top of its existing Experimentation Platform. Integrated directly into the Grab-Kit server middleware, the framework intercepts incoming requests and evaluates whether to inject failures using local variable resolution. Supported failure primitives include latency, errors, panics, rate throttling, and resource leaks to test dependent services. Combining chaos testing with experimentation telemetry enables engineers to correlate injected infrastructure disruptions with business metric impacts.

## Context

Hundreds of distributed microservices run at Grab, where unhandled failures in non-critical services during RPC calls could lead to full outages if contingency strategies like retries and circuit breakers are misconfigured.

## Approach / What changed

Building a Chaos SDK and dedicated UI on top of Grab's Experimentation Platform (ExP) and embedding it within the Grab-Kit server middleware to inject controlled application-level failure modes during runtime.

## Takeaways

- Integrating the Chaos SDK into Grab-Kit middleware enables automatic failure evaluation on incoming requests without requiring engineers to modify individual endpoint handlers.
- The underlying Experimentation Platform SDK resolves failure variables in single-digit microseconds locally without making external network calls.
- Chaos ExP supports eight failure primitives: errors, CPU load, memory leaks, latency, disk space consumption, goroutine leaks, panics, and request throttling.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Microservices](https://yomu.fyi/topic/microservices), [Reliability](https://yomu.fyi/topic/reliability), [Testing](https://yomu.fyi/topic/testing)

[Read original post](https://engineering.grab.com/chaos-engineering)