Scaling Like a Boss with Presto

Summary

Grab experienced severe performance degradation, long queue times, and connection timeouts on its Amazon Redshift analytics cluster as user concurrency and reporting workloads expanded. Although an initial Amazon S3 data lake decoupled storage from compute, business users required standard SQL interfaces rather than Spark data pipelines. The team deployed Presto clusters on AWS EMR, switching their storage format from AVRO to Parquet to support ANSI SQL querying directly against S3. Utilizing a shared Hive metastore on Amazon RDS allowed Grab to adopt a shared-data multi-cluster architecture that isolated distinct workloads across dedicated compute clusters. This setup enabled rapid cluster scaling, streamlined failover, and matched Redshift performance on partitioned time-range queries.

Context

Grab's Amazon Redshift analytics cluster struggled to handle growing query workloads, report concurrency, and peak demand periods, leading to long wait times, connection timeouts, and operational bottlenecks.

Approach / What changed

Grab implemented an S3-based data lake with Parquet storage and deployed isolated Presto clusters on AWS EMR connected to a unified Hive metastore on Amazon RDS.

Takeaways

• Adopting a shared-data multi-cluster Presto architecture over S3 enables workload isolation, rapid horizontal scaling, and standby cluster failovers without replicating data.
• Switching storage from AVRO to Parquet with granular time-based S3 partitions allowed Presto to achieve query speeds comparable to Redshift for time-bounded analytical workloads.
• Updating mutable datasets within the S3 data lake required a recursive data processing (RDP) pattern using Spark jobs orchestrated by an in-house dependency scheduler.

Related reading

• Grab · Jul 10, 2026

  Scaling Grab's Data Lake: Our journey to Apache Iceberg adoption

  Grab transitioned its petabyte-scale data lake from Hive Parquet to Apache Iceberg to resolve metadata bottlenecks, small file fragmentation, and lack of native ACID support. Implementing Iceberg with Z-ordering reduced navigation query runtimes from 70 seconds to 6 seconds and cut daily S3 API costs by up to 95% on key operational datasets. To seamlessly manage concurrent table formats without forcing engineers to specify format prefixes, Grab developed and open-sourced UnifiedSparkCatalog. This custom catalog transparently detects table types via the Hive Metastore, routes queries to format-specific handlers, and retains backwards compatibility for legacy Hive operations.

  Rahul Penti

• Grab · Feb 23, 2024

  Enabling near real-time data analytics on the data lake

  Traditional data lake setups using Parquet on Hive metastores struggle with frequent updates and long pipeline intervals, introducing significant latency for ad hoc queries. Grab solved this by implementing Apache Hudi to support near real-time analytics across bounded relational databases and unbounded Kafka streams. For high-throughput sources, Flink streams Avro log files to Merge On Read tables and generates compaction plans for asynchronous Spark writers. Low-throughput workloads leverage Copy On Write tables, while relational database sources ingest binlogs via Flink Change Data Capture connectors. This architecture reduced analytics data latency to the minute level without overloading production databases and Kafka clusters.

  Shi Kai Ng

• Grab · Mar 3, 2019

  How We Simplified Our Data Ingestion & Transformation Process

  Grab evolved its real-time data ingestion pipeline after an initial architecture built on Spark Streaming and Python encountered operational complexity, node failures, and data loss from S3 eventual consistency. Because the streaming workload primarily handled event partitioning and ORC file generation, the team consolidated these tasks directly into an existing Golang processing service. They implemented sharded concurrent maps for high-throughput partitioning and optimized heap allocations to resolve memory bottlenecks. This refactor removed intermediate Avro conversions and intermediate storage hops. The simplified Go pipeline eliminated data loss and reduced processing lag from up to 13 minutes down to approximately 1 minute.

  Yichao Wang

• Grab · Jan 2, 2019

  Querying Big Data in Real-time with Presto & Grab's TalariaDB

  Grab developed TalariaDB to support real-time SQL querying over high-velocity event streams while maintaining predictable sub-second latencies and low infrastructure costs. The distributed time-series store retains only the most recent hour of data and integrates directly with Presto via its PrestoThriftService interface. Internally, TalariaDB uses the Go-based Badger key-value store to maintain an in-memory key index of metric names and timestamps while mapping columnar event payloads directly to disk. Ingestion occurs by processing pre-partitioned event batches written to Amazon S3 via SQS notifications. By combining a zero-copy decoder with parallel split evaluation across gossiping cluster nodes, the architecture scales horizontally while serving millions of events per second.

  Roman Atachiants