Uncovering the Truth Behind Lua and Redis Data Consistency

Summary

Grab experienced replica CPU usage spikes following service deployments in their master/replica Redis cluster, which caused failovers to spike to 100% CPU. Investigation revealed that a post-deployment Lua monitor script executed separately on both nodes and relied on non-deterministic HGETALL key ordering. Redis encodes hash objects as either ziplists or hashtables, and restoring from an RDB snapshot initializes small hashes as ziplists even if the master previously converted them to hashtables. This encoding discrepancy caused key ordering to diverge, preventing secondary data from deleting correctly and bloating dataset sizes. Grab resolved the issue by sorting the outputs of HKEYS and HGETALL within the Lua script to guarantee deterministic execution across nodes.

Takeaways

1. 1

   Redis executes Lua scripts independently on replicas by default rather than replicating the resulting write commands, risking data divergence if scripts are non-deterministic.

2. 2

   Hash object encoding is not guaranteed to be identical between master and replica because RDB snapshot restoration initializes hashes as ziplists regardless of prior master-side conversions to hashtables.

3. 3

   Sorting HKEYS and HGETALL outputs inside Lua scripts provides deterministic execution with low overhead for small hash structures without increasing replication network traffic.