A Key Expired in Redis, You Won't Believe What Happened Next

Summary

Grab experienced an issue where its Unicorn API served stale data for up to 45 to 60 minutes despite expected cache invalidation times totaling around 11 minutes. The setup utilized ElastiCache Redis 2.x configured with a single master node for writes and two read-only slaves handling reads. Investigation revealed that in Redis 2.x, slave nodes do not expire keys on their own and only delete them upon receiving an explicit DEL command from the master. Because the master only actively checks and deletes 200 random keys per second, clearing expired keys across roughly 5.6 million cached items mathematically required over 110 hours, resulting in slaves serving expired data.

Takeaways

1. 1

   In Redis 2.x, read-only slave nodes cannot expire keys independently and rely entirely on DEL commands propagated from the master node.

2. 2

   Redis actively tests only 20 random keys with an expire set 10 times per second on the master, which causes significant delays in clearing expired keys across multi-million key datasets.

3. 3

   Relying solely on slave nodes for reads in Redis 2.x risks serving stale data for keys that have expired on the slave but have not yet been evaluated and deleted by the master.