Querying Big Data in Real-time with Presto & Grab's TalariaDB

Summary

Grab developed TalariaDB to support real-time SQL querying over high-velocity event streams while maintaining predictable sub-second latencies and low infrastructure costs. The distributed time-series store retains only the most recent hour of data and integrates directly with Presto via its PrestoThriftService interface. Internally, TalariaDB uses the Go-based Badger key-value store to maintain an in-memory key index of metric names and timestamps while mapping columnar event payloads directly to disk. Ingestion occurs by processing pre-partitioned event batches written to Amazon S3 via SQS notifications. By combining a zero-copy decoder with parallel split evaluation across gossiping cluster nodes, the architecture scales horizontally while serving millions of events per second.

Takeaways

1. 1

   TalariaDB uses Badger's key-value store to keep lexicographically ordered metric and timestamp keys in memory while memory-mapping columnar event batches for zero-copy retrieval.

2. 2

   Ingestion delegates partitioning and batch creation to upstream pipeline services, allowing TalariaDB to ingest files directly from S3 upon receiving SQS notifications.

3. 3

   Presto queries TalariaDB instances in parallel using the Presto Thrift Connector, while nodes maintain cluster membership via the Gossip protocol and register in Route 53.