Scaling Like a Boss with Presto

Summary

Grab experienced severe performance degradation, long queue times, and connection timeouts on its Amazon Redshift analytics cluster as user concurrency and reporting workloads expanded. Although an initial Amazon S3 data lake decoupled storage from compute, business users required standard SQL interfaces rather than Spark data pipelines. The team deployed Presto clusters on AWS EMR, switching their storage format from AVRO to Parquet to support ANSI SQL querying directly against S3. Utilizing a shared Hive metastore on Amazon RDS allowed Grab to adopt a shared-data multi-cluster architecture that isolated distinct workloads across dedicated compute clusters. This setup enabled rapid cluster scaling, streamlined failover, and matched Redshift performance on partitioned time-range queries.

Takeaways

1. 1

   Adopting a shared-data multi-cluster Presto architecture over S3 enables workload isolation, rapid horizontal scaling, and standby cluster failovers without replicating data.

2. 2

   Switching storage from AVRO to Parquet with granular time-based S3 partitions allowed Presto to achieve query speeds comparable to Redshift for time-bounded analytical workloads.

3. 3

   Updating mutable datasets within the S3 data lake required a recursive data processing (RDP) pattern using Spark jobs orchestrated by an in-house dependency scheduler.