How we store and process millions of orders daily

Summary

The Grab Order Platform processes millions of food and mart transactions daily, requiring high throughput, fault tolerance, and reduced cloud costs across transactional and analytical workloads. To meet these demands, the engineering team decoupled their database architecture by using Amazon DynamoDB for critical OLTP queries and MySQL RDS for historical OLAP queries. DynamoDB handles online order lifecycles with strong consistency, utilizing sparse Global Secondary Indexes for ongoing orders and TTL configurations to limit storage growth. Updates propagate asynchronously to MySQL RDS through a Kafka ingestion pipeline backed by Amazon SQS retries and timestamp-based version checks. This dual-database approach isolated core transaction availability from analytical queries and delivered significant cloud cost savings.

Context

Grab's Order Platform needed to handle high read and write QPS during peak hours, support distinct transactional and analytical query patterns, maintain high availability for online ordering, and reduce cloud costs as order volume scaled.

Approach / What changed

Decoupling transactional and analytical workloads by serving online order processing through DynamoDB and analytical queries through MySQL RDS, connected by an asynchronous Kafka ingestion pipeline with SQS fallbacks and strict data retention policies.

Takeaways

• DynamoDB handles OLTP queries using sparse Global Secondary Indexes that automatically delete entries when order status updates clear the index key, keeping ongoing order lookups compact.
• The data ingestion pipeline pairs Kafka with Amazon SQS fallback queues, database unique keys, and microsecond-precision update timestamps to handle retries, deduplication, and out-of-order events.
• Data retention policies reduce storage costs by dropping monthly MySQL partitions older than six months and setting DynamoDB TTL expirations to three months.

Related reading

• Grab · Jul 15, 2024

  LLM-powered data classification for data entities at scale

  Grab needed to classify sensitive data at the table and column level across petabytes of database tables and streaming schemas. Manual schema-tiering campaigns had resulted in half of all schemas receiving overly strict Tier 1 access controls, while an initial automated service using regex patterns and third-party machine learning produced high false-positive rates and lacked customizability. To address this, the Caspian data engineering and governance teams enhanced their internal orchestration service, Gemini, by integrating GPT-3.5 via Azure OpenAI. Gemini aggregates classification requests into mini-batches, handles API rate limits, and uses prompt engineering—including few-shot examples, curated tag libraries, and explicit JSON DTO schemas—to reliably tag columns for data owner verification.

  Hualin Liu

• Grab · Dec 26, 2023

  Kafka on Kubernetes: Reloaded for fault tolerance

  Grab's real-time data streaming platform, Coban, operates Kafka on AWS Elastic Kubernetes Service using Strimzi, allocating an entire EC2 worker node with NVMe instance store volumes to each broker. An initial architecture suffered from client connection errors, broken Network Load Balancer target groups, and zombie Persistent Volume Claims when worker nodes terminated. To achieve automated fault tolerance, the team integrated the AWS Node Termination Handler in Queue Processor mode with Auto Scaling lifecycle hooks, ensuring Kafka receives a SIGTERM to migrate partition leadership gracefully before shutdown. They also introduced the Kubernetes Cluster Autoscaler to dynamically provision replacement nodes during maintenance events and used the AWS Load Balancer Controller with TargetGroupBinding custom resources to dynamically update load balancer targets using IP mode.

  Fabrice Harbulot

• Grab · Aug 5, 2024

  How we improved translation experience with cost efficiency

  Grab faced translation quality and cost challenges in its booking chat system when tourism resumed after COVID restrictions eased. Device language settings proved unreliable for determining message language, prompting the team to use Lingua alongside booking context heuristics and third-party fallbacks for real-time detection. To replace expensive and inaccurate general translation APIs, Grab distilled a large open-source Hugging Face model into lightweight, language-specific models trained on ten million synthetic chat examples. A post-translation validation step verifies non-translatable entities such as numbers and emojis before falling back to external services if mutations occur. Caching layers were also introduced across translation paths to curb redundant on-the-fly execution.

  Jie Zhang

• Grab · Aug 15, 2023

  Streamlining Grab's Segmentation Platform with faster creation and lower latency

  Grab's Segmentation Platform previously stored user-to-segment mappings across individual rows in ScyllaDB, causing write bottlenecks during segment creation and read latencies too high for downstream consumers. To resolve these performance limitations, the team transitioned to storing segments as Roaring Bitmaps saved as single blobs in object storage. This compression strategy splits 32-bit integer user IDs into chunks across array, bitmap, and run containers based on data density, reducing a one-million-member segment to under one megabyte. An accompanying client SDK manages segment retrieval, decoding, update notifications, and least-recently-used in-memory caching. Consequently, consumers such as Grab's communications platform achieved peak throughput of 15,000 queries per second with sub-millisecond p99 read latencies.

  Jake Ng