# How we store and process millions of orders daily

[Grab](https://yomu.fyi/company/grab) · Xi Chen · Aug 15, 2022

**Type:** Problem & solution

## Summary

The Grab Order Platform processes millions of food and mart transactions daily, requiring high throughput, fault tolerance, and reduced cloud costs across transactional and analytical workloads. To meet these demands, the engineering team decoupled their database architecture by using Amazon DynamoDB for critical OLTP queries and MySQL RDS for historical OLAP queries. DynamoDB handles online order lifecycles with strong consistency, utilizing sparse Global Secondary Indexes for ongoing orders and TTL configurations to limit storage growth. Updates propagate asynchronously to MySQL RDS through a Kafka ingestion pipeline backed by Amazon SQS retries and timestamp-based version checks. This dual-database approach isolated core transaction availability from analytical queries and delivered significant cloud cost savings.

## Context

Grab's Order Platform needed to handle high read and write QPS during peak hours, support distinct transactional and analytical query patterns, maintain high availability for online ordering, and reduce cloud costs as order volume scaled.

## Approach / What changed

Decoupling transactional and analytical workloads by serving online order processing through DynamoDB and analytical queries through MySQL RDS, connected by an asynchronous Kafka ingestion pipeline with SQS fallbacks and strict data retention policies.

## Takeaways

- DynamoDB handles OLTP queries using sparse Global Secondary Indexes that automatically delete entries when order status updates clear the index key, keeping ongoing order lookups compact.
- The data ingestion pipeline pairs Kafka with Amazon SQS fallback queues, database unique keys, and microsecond-precision update timestamps to handle retries, deduplication, and out-of-order events.
- Data retention policies reduce storage costs by dropping monthly MySQL partitions older than six months and setting DynamoDB TTL expirations to three months.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [AWS](https://yomu.fyi/topic/aws), [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Kafka](https://yomu.fyi/topic/kafka), [MySQL](https://yomu.fyi/topic/mysql)

[Read original post](https://engineering.grab.com/how-we-store-millions-orders)