# Querying Big Data in Real-time with Presto & Grab's TalariaDB

[Grab](https://yomu.fyi/company/grab) · Roman Atachiants · Jan 2, 2019

**Type:** Problem & solution

## Summary

Grab developed TalariaDB to support real-time SQL querying over high-velocity event streams while maintaining predictable sub-second latencies and low infrastructure costs. The distributed time-series store retains only the most recent hour of data and integrates directly with Presto via its PrestoThriftService interface. Internally, TalariaDB uses the Go-based Badger key-value store to maintain an in-memory key index of metric names and timestamps while mapping columnar event payloads directly to disk. Ingestion occurs by processing pre-partitioned event batches written to Amazon S3 via SQS notifications. By combining a zero-copy decoder with parallel split evaluation across gossiping cluster nodes, the architecture scales horizontally while serving millions of events per second.

## Context

Grab needed to query at least 2 to 3 terabytes of high-velocity real-time event data per hour using SQL with predictable low latency and low cost, to enable fast decision-making when experiments impact platform metrics.

## Approach / What changed

Grab built TalariaDB, a distributed time-series data store written in Go that ingests columnar event batches from S3 via SQS notifications, stores them in Badger LSM-trees with time-based eviction, and exposes data to Presto via the Presto Thrift Connector.

## Takeaways

- TalariaDB uses Badger's key-value store to keep lexicographically ordered metric and timestamp keys in memory while memory-mapping columnar event batches for zero-copy retrieval.
- Ingestion delegates partitioning and batch creation to upstream pipeline services, allowing TalariaDB to ingest files directly from S3 upon receiving SQS notifications.
- Presto queries TalariaDB instances in parallel using the Presto Thrift Connector, while nodes maintain cluster membership via the Gossip protocol and register in Route 53.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Go](https://yomu.fyi/topic/go), [Scalability](https://yomu.fyi/topic/scalability), [Streaming](https://yomu.fyi/topic/streaming)

[Read original post](https://engineering.grab.com/big-data-real-time-presto-talariadb)