Dealing with the Meltdown Patch at Grab

Summary

AWS infrastructure maintenance related to Meltdown patches led to severe CPU utilization spikes across Grab's ElastiCache Redis instances. Because Redis is single-threaded, spikes past 50% CPU on two-vCPU instances threatened service capacity, and initial Multi-AZ failovers only provided temporary relief until the new master nodes received rolling patches. To handle the increased overhead before their peak traffic window, the engineering team horizontally scaled both clustered and non-clustered Redis fleets. For Redis 3.2.4 clusters lacking live re-sharding support, they provisioned larger clusters, warmed caches, and redirected traffic. Non-clustered workloads were resolved by provisioning extra nodes, migrating compatible services to Redis Cluster, or updating application code to shard data across multiple instances.

Context

Rolling Meltdown patches deployed to AWS ElastiCache caused sudden, significant CPU utilization spikes across hundreds of Grab's Redis nodes, pushing single-threaded workloads past safe operational thresholds.

Approach / What changed

Grab spun up new Redis 3.2.4 clusters with additional shards, warmed caches, and migrated traffic away from old clusters, while resolving non-clustered instances by adding nodes, adopting Redis Cluster, or updating application code to shard data.

Takeaways

• Single-threaded Redis running on a dual-vCPU instance reaches maximum single-core capacity when total instance CPU utilization exceeds 50%.
• Multi-AZ failovers only temporarily alleviated CPU pressure because replica instances experienced identical spikes once AWS applied rolling Meltdown patches to them.
• Because Redis engine version 3.2.4 lacked live re-sharding support, adding shards required provisioning new clusters, warming the cache, and decommissioning old infrastructure.

Related reading

• Grab · Aug 8, 2017

  Migrating Existing Datastores

  Grab's Identity team faced imminent memory exhaustion on a single Redis node used to cache mobile authentication tokens under rapid user growth. Because read traffic outweighed write traffic by roughly 200 times, the team opted for an AWS ElastiCache cluster with three shards and two read replicas per shard. They executed a zero-downtime, six-phase migration plan while handling a peak load of 20,000 queries per second. The migration transitioned through initial one-time data replication, asynchronous shadow writes, synchronous dual writes, asynchronous read validation, switching primary reads, and final write cleanup. Controlled by feature flags and monitored with metrics at every stage, the migration completed without invalidating tokens or causing service disruptions.

  Nishant Gupta

• Grab · May 5, 2019

  Preventing Pipeline Calls from Crashing Redis Clusters

  A single Redis slave node failure caused Grab's Apollo booking service to suffer an over 95 percent call failure rate for one minute despite running a three-shard cluster with two replicas per partition. Investigation revealed that the service configured Go-Redis to route all read queries exclusively to slave nodes to offload master CPU usage. When the slave node dropped offline, batched HMGET pipeline calls failed completely because the client wrapper treated a single command failure as a failure of the entire pipeline. Furthermore, the Go-Redis client cached cluster topology and only lazily refreshed state every sixty seconds, continuing to direct traffic to the dead replica until the timer expired. Grab addressed this risk by recommending dedicated pipeline clients configured with latency-based routing to allow reads to fall back to responsive master nodes.

  Michael Cartmell

• Grab · Sep 7, 2020

  Uncovering the Truth Behind Lua and Redis Data Consistency

  Grab experienced replica CPU usage spikes following service deployments in their master/replica Redis cluster, which caused failovers to spike to 100% CPU. Investigation revealed that a post-deployment Lua monitor script executed separately on both nodes and relied on non-deterministic HGETALL key ordering. Redis encodes hash objects as either ziplists or hashtables, and restoring from an RDB snapshot initializes small hashes as ziplists even if the master previously converted them to hashtables. This encoding discrepancy caused key ordering to diverge, preventing secondary data from deleting correctly and bloating dataset sizes. Grab resolved the issue by sorting the outputs of HKEYS and HGETALL within the Lua script to guarantee deterministic execution across nodes.

  Allen Wang

• Grab · Jan 16, 2019

  A Lean and Scalable Data Pipeline to Capture Large Scale Events and Support Experimentation Platform

  Controlled online experimentation across diverse product verticals requires tracking interactions across systems to prevent local optimizations from causing global degradation. Grab built a batch data pipeline to capture, ingest, and process petabytes of event data to support its experimentation platform and analytics stakeholders. The architecture loads ingested event data from Amazon S3, transforms and sorts it, and writes partitioned output back to S3 with metadata registered in Apache Hive. Using Apache Spark on AWS Elastic MapReduce with Apache Airflow for orchestration, the system handles roughly 400,000 incoming events per second. The data is partitioned by event type and ingestion time and stored in Apache ORC format to streamline query workloads and reduce retrieval overhead.

  Oscar Cassetti