# Turbocharging GrabUnlimited with Temporal

[Grab](https://yomu.fyi/company/grab) · Michel Parreno · Dec 12, 2024

## Summary

GrabUnlimited experienced scaling bottlenecks, corrupted membership states, and elevated production incidents after its subscriber base grew by over 1000%. The original architecture relied on Amazon SQS state machines, 5-minute Redis locks, and daily batch cron jobs that overwhelmed the database and lacked granular idempotency during upstream retries. To eliminate these failure modes, the engineering team migrated the core membership lifecycle to Temporal's workflow orchestration engine. Replacing batch cron jobs with Temporal Timers distributed renewal operations throughout the day, while matching workflow IDs prevented race conditions between renewals and cancellations. This architectural transition resolved database bottlenecks and yielded an 80% reduction in open production incidents.

## Takeaways

- Retrying multi-step SQS state handlers generated new idempotency keys upon failure, causing duplicate benefit disbursements unless states were split into hundreds of micro-steps.
- Replacing daily batch renewal cron jobs with Temporal Timers distributed renewals across the entire day based on exact subscription times, removing the renewal database bottleneck.
- Assigning identical workflow IDs to mutually exclusive processes, such as membership cancellation and auto-renewal, prevented concurrency race conditions and eliminated zombie states caused by expiring Redis locks.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Migrations](https://yomu.fyi/topic/migration), [Reliability](https://yomu.fyi/topic/reliability), [Scalability](https://yomu.fyi/topic/scalability)

[Read original post](https://engineering.grab.com/turbocharging-grabunlimited-with-temporal)