# Dealing with the Meltdown Patch at Grab

[Grab](https://yomu.fyi/company/grab) · Althaf Hameez · Jan 7, 2018

**Type:** Incident / postmortem

## Summary

AWS infrastructure maintenance related to Meltdown patches led to severe CPU utilization spikes across Grab's ElastiCache Redis instances. Because Redis is single-threaded, spikes past 50% CPU on two-vCPU instances threatened service capacity, and initial Multi-AZ failovers only provided temporary relief until the new master nodes received rolling patches. To handle the increased overhead before their peak traffic window, the engineering team horizontally scaled both clustered and non-clustered Redis fleets. For Redis 3.2.4 clusters lacking live re-sharding support, they provisioned larger clusters, warmed caches, and redirected traffic. Non-clustered workloads were resolved by provisioning extra nodes, migrating compatible services to Redis Cluster, or updating application code to shard data across multiple instances.

## Context

Rolling Meltdown patches deployed to AWS ElastiCache caused sudden, significant CPU utilization spikes across hundreds of Grab's Redis nodes, pushing single-threaded workloads past safe operational thresholds.

## Approach / What changed

Grab spun up new Redis 3.2.4 clusters with additional shards, warmed caches, and migrated traffic away from old clusters, while resolving non-clustered instances by adding nodes, adopting Redis Cluster, or updating application code to shard data.

## Takeaways

- Single-threaded Redis running on a dual-vCPU instance reaches maximum single-core capacity when total instance CPU utilization exceeds 50%.
- Multi-AZ failovers only temporarily alleviated CPU pressure because replica instances experienced identical spikes once AWS applied rolling Meltdown patches to them.
- Because Redis engine version 3.2.4 lacked live re-sharding support, adding shards required provisioning new clusters, warming the cache, and decommissioning old infrastructure.

**Tags:** [AWS](https://yomu.fyi/topic/aws), [Caching](https://yomu.fyi/topic/caching), [Incident Response](https://yomu.fyi/topic/incident-response), [Performance](https://yomu.fyi/topic/performance), [Redis](https://yomu.fyi/topic/redis)

[Read original post](https://engineering.grab.com/dealing-with-the-meltdown-patch-at-grab)