Uncovering the Truth Behind Lua and Redis Data Consistency

Summary

Grab experienced replica CPU usage spikes following service deployments in their master/replica Redis cluster, which caused failovers to spike to 100% CPU. Investigation revealed that a post-deployment Lua monitor script executed separately on both nodes and relied on non-deterministic HGETALL key ordering. Redis encodes hash objects as either ziplists or hashtables, and restoring from an RDB snapshot initializes small hashes as ziplists even if the master previously converted them to hashtables. This encoding discrepancy caused key ordering to diverge, preventing secondary data from deleting correctly and bloating dataset sizes. Grab resolved the issue by sorting the outputs of HKEYS and HGETALL within the Lua script to guarantee deterministic execution across nodes.

Context

Deployments caused Redis replica CPU usage to spike higher than the master, and a subsequent failover caused the new master to immediately hit 100% CPU capacity with read traffic.

Approach / What changed

Grab made the Lua monitor script deterministic by sorting the outputs of HKEYS and HGETALL commands, avoiding the alternative of enabling script effect replication.

Takeaways

• Redis executes Lua scripts independently on replicas by default rather than replicating the resulting write commands, risking data divergence if scripts are non-deterministic.
• Hash object encoding is not guaranteed to be identical between master and replica because RDB snapshot restoration initializes hashes as ziplists regardless of prior master-side conversions to hashtables.
• Sorting HKEYS and HGETALL outputs inside Lua scripts provides deterministic execution with low overhead for small hash structures without increasing replication network traffic.

Related reading

• Grab · Apr 19, 2021

  How We Improved Agent Chat Efficiency with Machine Learning

  Agent typing time represented a large portion of Grab's chat support journey, and 85% of messages were still free typed because agents customized static templates to fit their personal style. To accelerate typing across multilingual markets without robotic templates, Grab built SmartChat, a machine learning feature that provides contextual sentence completion. The team opted for a lightweight seq2seq architecture using single-layered GRU encoder-decoders in TensorFlow instead of bulky attention models to keep model latency under 100ms. The user interface was implemented in React using a content-editable div with inline typeahead suggestions activated via keyboard shortcuts.

  Suman Anand

• Grab · Jul 13, 2021

  App Modularisation at Scale

  Grab transitioned its monolithic mobile application into a modular architecture to resolve increasing code conflicts, slow releases, and difficult team collaboration. The team decomposed the single module by establishing base infrastructure modules, shared UI and utility libraries, discrete feature modules, and bridge kit modules for inter-module communication. Dependency injection using Dagger ties these components together in the main app module while preventing feature modules from directly depending on one another. The architecture spans over 1,000 modules across the app, with more than 200 modules in the Grab Financial Group payments domain where over 95% of modules build in under 15 seconds. This approach accelerated Gradle CI and local builds through parallel compilation and caching, though it increased Gradle sync times, IDE memory usage, and configuration maintenance overhead.

  Amar Jain

• Grab · Jul 29, 2021

  How We Cut GrabFood.com’s Page JavaScript Asset Sizes by 3x

  GrabFood.com faced high cloud costs while serving over 1 TB of weekly network egress across 175 million requests. To minimize egress and improve page performance, the engineering team audited and reduced their webpack dependencies using tools such as webpack-bundle-analyzer and dependency-cruiser. Their strategies included lazy loading rarely used libraries, unifying duplicate modules under single entry points, and substituting utility libraries like axios with native browser Web APIs. Additionally, altering implementation approaches, such as replacing client-side signed JWT cookie encoding with plain JSON strings, eliminated heavy Node dependencies like crypto. These optimizations reduced JavaScript static assets from 750 KB to 250 KB, decreased CloudFront costs by 20%, and accelerated build times by 3.6x.

  Gibson Cheng

• Grab · Dec 3, 2020

  How Grab is Blazing Through the Superapp Bazel Migration

  Grab's mobile superapp scaled past 2.5 million lines of code across both Android and iOS, leading to unsustainable local and CI build times under Gradle and Xcode. To address these bottlenecks, the engineering team analyzed their dependency trees and introduced an internal tool to calculate and optimize the build critical path. They also deployed a Kubernetes-autoscaled remote build system using Mainframer for Android and implemented Test Impact Analysis to run only affected tests in pre-merge validation. While dependency decoupling yielded modest 7% to 10% gains and iOS remote builds proved unscalable on Apple hardware, Android remote builds reduced local compile times by up to 50%, and targeted test execution reduced pre-merge pipeline durations by more than 30%.

  Sergii Grechukha