# Preventing Pipeline Calls from Crashing Redis Clusters

[Grab](https://yomu.fyi/company/grab) · Michael Cartmell · May 5, 2019

**Type:** Incident / postmortem

## Summary

A single Redis slave node failure caused Grab's Apollo booking service to suffer an over 95 percent call failure rate for one minute despite running a three-shard cluster with two replicas per partition. Investigation revealed that the service configured Go-Redis to route all read queries exclusively to slave nodes to offload master CPU usage. When the slave node dropped offline, batched HMGET pipeline calls failed completely because the client wrapper treated a single command failure as a failure of the entire pipeline. Furthermore, the Go-Redis client cached cluster topology and only lazily refreshed state every sixty seconds, continuing to direct traffic to the dead replica until the timer expired. Grab addressed this risk by recommending dedicated pipeline clients configured with latency-based routing to allow reads to fall back to responsive master nodes.

## Context

A single Redis slave node failure in Grab's Apollo booking service caused an unexpected 95+% call failure rate and a one-minute partial outage, breaching the intended fault tolerance of its multi-replica Redis Cluster.

## Approach / What changed

Grab analyzed the Go-Redis client library and internal wrappers, identifying that lazy one-minute topology refreshes and strict slave-only read routing broke pipelined requests, and recommended configuring a dedicated client for pipelining with RouteByLatency enabled.

## Takeaways

- A single failed command inside a Go-Redis pipeline batch caused Grab's wrapper function to treat the entire multi-command pipeline execution as failed.
- The Go-Redis client lazily refreshed its cluster topology cache only after one minute, maintaining traffic to unreachable slave nodes throughout that window.
- Enabling RouteByLatency directs pipelined reads to master nodes when ping latency is under 1ms, trading master CPU usage for improved resilience against replica failures.

**Tags:** [Go](https://yomu.fyi/topic/go), [Incident Response](https://yomu.fyi/topic/incident-response), [Redis](https://yomu.fyi/topic/redis), [Reliability](https://yomu.fyi/topic/reliability)

[Read original post](https://engineering.grab.com/preventing-pipeline-calls-from-crashing-redis-clusters)