Preventing Pipeline Calls from Crashing Redis Clusters

Summary

A single Redis slave node failure caused Grab's Apollo booking service to suffer an over 95 percent call failure rate for one minute despite running a three-shard cluster with two replicas per partition. Investigation revealed that the service configured Go-Redis to route all read queries exclusively to slave nodes to offload master CPU usage. When the slave node dropped offline, batched HMGET pipeline calls failed completely because the client wrapper treated a single command failure as a failure of the entire pipeline. Furthermore, the Go-Redis client cached cluster topology and only lazily refreshed state every sixty seconds, continuing to direct traffic to the dead replica until the timer expired. Grab addressed this risk by recommending dedicated pipeline clients configured with latency-based routing to allow reads to fall back to responsive master nodes.

Context

A single Redis slave node failure in Grab's Apollo booking service caused an unexpected 95+% call failure rate and a one-minute partial outage, breaching the intended fault tolerance of its multi-replica Redis Cluster.

Approach / What changed

Grab analyzed the Go-Redis client library and internal wrappers, identifying that lazy one-minute topology refreshes and strict slave-only read routing broke pipelined requests, and recommended configuring a dedicated client for pipelining with RouteByLatency enabled.

Takeaways

• A single failed command inside a Go-Redis pipeline batch caused Grab's wrapper function to treat the entire multi-command pipeline execution as failed.
• The Go-Redis client lazily refreshed its cluster topology cache only after one minute, maintaining traffic to unreachable slave nodes throughout that window.
• Enabling RouteByLatency directs pipelined reads to master nodes when ping latency is under 1ms, trading master CPU usage for improved resilience against replica failures.

Related reading

• Grab · Jan 7, 2018

  Dealing with the Meltdown Patch at Grab

  AWS infrastructure maintenance related to Meltdown patches led to severe CPU utilization spikes across Grab's ElastiCache Redis instances. Because Redis is single-threaded, spikes past 50% CPU on two-vCPU instances threatened service capacity, and initial Multi-AZ failovers only provided temporary relief until the new master nodes received rolling patches. To handle the increased overhead before their peak traffic window, the engineering team horizontally scaled both clustered and non-clustered Redis fleets. For Redis 3.2.4 clusters lacking live re-sharding support, they provisioned larger clusters, warmed caches, and redirected traffic. Non-clustered workloads were resolved by provisioning extra nodes, migrating compatible services to Redis Cluster, or updating application code to shard data across multiple instances.

  Althaf Hameez

• Grab · Aug 8, 2017

  Migrating Existing Datastores

  Grab's Identity team faced imminent memory exhaustion on a single Redis node used to cache mobile authentication tokens under rapid user growth. Because read traffic outweighed write traffic by roughly 200 times, the team opted for an AWS ElastiCache cluster with three shards and two read replicas per shard. They executed a zero-downtime, six-phase migration plan while handling a peak load of 20,000 queries per second. The migration transitioned through initial one-time data replication, asynchronous shadow writes, synchronous dual writes, asynchronous read validation, switching primary reads, and final write cleanup. Controlled by feature flags and monitored with metrics at every stage, the migration completed without invalidating tokens or causing service disruptions.

  Nishant Gupta

• Grab · Sep 7, 2020

  Uncovering the Truth Behind Lua and Redis Data Consistency

  Grab experienced replica CPU usage spikes following service deployments in their master/replica Redis cluster, which caused failovers to spike to 100% CPU. Investigation revealed that a post-deployment Lua monitor script executed separately on both nodes and relied on non-deterministic HGETALL key ordering. Redis encodes hash objects as either ziplists or hashtables, and restoring from an RDB snapshot initializes small hashes as ziplists even if the master previously converted them to hashtables. This encoding discrepancy caused key ordering to diverge, preventing secondary data from deleting correctly and bloating dataset sizes. Grab resolved the issue by sorting the outputs of HKEYS and HGETALL within the Lua script to guarantee deterministic execution across nodes.

  Allen Wang

• Grab · Apr 11, 2025

  Evaluating performance impact of removing Redis-cache from a Scylla-backed service

  Grab operates a high-throughput Rust read service that aggregates counter metrics from Scylla tables across minutely, hourly, and daily granularities. The service initially cached aggregated responses in Redis using keys rounded to 15-minute intervals alongside a five-minute TTL. Because incoming queries predominantly requested recent data, transitions between 15-minute windows caused simultaneous cache misses across active configurations, resulting in severe Scylla traffic spikes, latency surges, and timeouts. To resolve the load imbalance, engineers proposed removing the Redis cache entirely and relying directly on Scylla's native internal caching. The rollout was staged in production by deterministically disabling Redis caching for specific counter configurations using mathematical operations on configuration IDs.

  Md Riyadh