Loading post…
GrabMichel ParrenoProblem & solution
Turbocharging GrabUnlimited with Temporal
Summary
Grab’s GrabUnlimited flagship membership program struggled as subscribers grew by over 1000% from January 2022 to June 2023. Its original design used a daily cron job for renewals, Amazon’s SQS to distribute state handlers, and five-minute Redis locks, but outages, corrupted states, concurrent cancellation and renewal, missed benefits, and non-idempotent retries created operational problems. The team migrated the service to a Temporal workflow-based system, enhancing its ability to handle millions of subscribers with greater efficiency and resilience. The source reports an 80% reduction in open production incidents and says the change improved the membership experience for users. This first part focuses on designing a robust and scalable membership system, while migration challenges and lessons are reserved for a subsequent installment.
Context
GrabUnlimited’s subscriber base increased by over 1000%, exposing low resiliency in a backend designed initially as a minimum viable product. Users experienced corrupted membership states, failed renewals, and missing benefits, while engineers spent one engineer per week handling incidents. The legacy architecture also had scalability, concurrency, resiliency, and idempotency problems.
Approach / What changed
The team migrated GrabUnlimited from its legacy state-machine system, which used a daily cron job, Amazon SQS, and five-minute Redis locks, to a Temporal workflow-based system. The migration was intended to improve the system’s efficiency and resilience while supporting millions of subscribers.
Takeaways
The legacy renewal design retrieved all memberships due each day through a cron job, creating database load surges that led to vertical scaling as subscriber numbers increased.
A five-minute Redis lock could expire during a renewal, allowing cancellation and renewal to run concurrently and leave a membership in a “zombie” state.
The Temporal migration is reported to have reduced open production incidents by 80% and improved the membership experience for users.
Source and freshness
- Source
- engineering.grab.com
- Published
- Added to Yomu
- Last updated
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.