Dealing with the Meltdown Patch at Grab

Summary

AWS infrastructure maintenance related to Meltdown patches led to severe CPU utilization spikes across Grab's ElastiCache Redis instances. Because Redis is single-threaded, spikes past 50% CPU on two-vCPU instances threatened service capacity, and initial Multi-AZ failovers only provided temporary relief until the new master nodes received rolling patches. To handle the increased overhead before their peak traffic window, the engineering team horizontally scaled both clustered and non-clustered Redis fleets. For Redis 3.2.4 clusters lacking live re-sharding support, they provisioned larger clusters, warmed caches, and redirected traffic. Non-clustered workloads were resolved by provisioning extra nodes, migrating compatible services to Redis Cluster, or updating application code to shard data across multiple instances.

Takeaways

1. 1

   Single-threaded Redis running on a dual-vCPU instance reaches maximum single-core capacity when total instance CPU utilization exceeds 50%.

2. 2

   Multi-AZ failovers only temporarily alleviated CPU pressure because replica instances experienced identical spikes once AWS applied rolling Meltdown patches to them.

3. 3

   Because Redis engine version 3.2.4 lacked live re-sharding support, adding shards required provisioning new clusters, warming the cache, and decommissioning old infrastructure.