# A Key Expired in Redis, You Won't Believe What Happened Next

[Grab](https://yomu.fyi/company/grab) · Karan Kamath · Mar 27, 2017

**Type:** Incident / postmortem

## Summary

Grab experienced an issue where its Unicorn API served stale data for up to 45 to 60 minutes despite expected cache invalidation times totaling around 11 minutes. The setup utilized ElastiCache Redis 2.x configured with a single master node for writes and two read-only slaves handling reads. Investigation revealed that in Redis 2.x, slave nodes do not expire keys on their own and only delete them upon receiving an explicit DEL command from the master. Because the master only actively checks and deletes 200 random keys per second, clearing expired keys across roughly 5.6 million cached items mathematically required over 110 hours, resulting in slaves serving expired data.

## Context

Grab's Unicorn API was serving region-specific data that was up to 60 minutes stale to consumers, despite multi-tier caches configured to hold data for at most 11 minutes.

## Approach / What changed

Engineers analyzed key TTL behavior on both master and slave nodes and evaluated the Redis 2.x replication and probabilistic active expiration algorithms against their dataset of roughly 5.6 million keys.

## Takeaways

- In Redis 2.x, read-only slave nodes cannot expire keys independently and rely entirely on DEL commands propagated from the master node.
- Redis actively tests only 20 random keys with an expire set 10 times per second on the master, which causes significant delays in clearing expired keys across multi-million key datasets.
- Relying solely on slave nodes for reads in Redis 2.x risks serving stale data for keys that have expired on the slave but have not yet been evaluated and deleted by the master.

**Tags:** [AWS](https://yomu.fyi/topic/aws), [Caching](https://yomu.fyi/topic/caching), [Redis](https://yomu.fyi/topic/redis), [Reliability](https://yomu.fyi/topic/reliability)

[Read original post](https://engineering.grab.com/a-key-expired-in-redis-you-wont-believe-what-happened-next)