Securing and Managing Multi-cloud Presto Clusters with Grab’s DataGateway

Summary

Grab's data engineering team needed to manage data access across growing Presto workloads spanning hundreds of users, thousands of tables, and multiple clouds without exposing infrastructure churn to clients. Out-of-the-box Presto lacked fine-grained schema- and table-level access controls, unified endpoint routing, and seamless cluster switching. To solve this, the team built DataGateway, a smart HTTP proxy and abstraction layer deployed as microservices on Kubernetes. The platform parses incoming SQL statements, verifies user access control lists against target schemas and tables, and dynamically routes queries to appropriate AWS or Azure Presto clusters without requiring endpoint updates. This architecture enabled zero-disruption cluster migrations across 40 clusters and unified access management across a multi-cloud data lake environment.

Context

Grab needed a single endpoint to serve hundreds of Presto users, enforce schema-, table-, and column-level access control beyond basic cloud IAM storage controls, capture audit trails, and maintain seamless user experiences during cluster provisioning, upgrades, and multi-cloud expansions across AWS and Azure.

Approach / What changed

Grab built DataGateway as a Kubernetes-hosted microservices platform comprising an API Service that mimics the Presto API, a version-matched SQL Parser to extract requested schemas and tables, an Auth Framework handling modular authentication and ACL checks, and an Admin UI for access and cluster management.

Takeaways

• AWS IAM roles on EMR/EKS provide only bucket- and file-level S3 control, making application-level SQL statement parsing necessary for schema-, table-, and field-level permissions.
• DataGateway compiles separate SQL parser engines to match each distinct running version of Presto because SQL parsing behaviors vary across releases.
• Using DataGateway as a proxy allowed Grab to migrate 40 Presto clusters on AWS EMR and route queries between AWS and Azure dynamically without requiring end users to change connection endpoints.

Related reading

• Grab · Jul 19, 2021

  Processing ETL tasks with Ratchet

  Grab's Lending team relies on multiple microservices to finance various user segments, requiring automated pipelines for operational tasks like loan rescheduling and merchant whitelisting. To execute these workflows, the team uses Ratchet, a Go library that organizes data pipelines into concurrent stages connected by Go channels and JSON payloads. In the Azkaban service, each ETL task is structured as a Job Processor that configures custom Data Processors for extraction, transformation, and loading. For merchant whitelisting, a custom processor pulls uploaded CSV files from Amazon S3, validates the rows to prevent manual errors, calls microservice APIs, and passes the output to an empty load stage. This pipeline architecture allows the team to isolate failures quickly, eliminate manual data validation, and automate routine operational tasks.

  Amar Prakash

• Grab · Oct 30, 2020

  Keeping 170 Libraries Up to Date on a Large Scale Android App

  Grab's passenger Android superapp relies on more than 170 in-house and open-source libraries, incorporating five to ten library bumps into each weekly release. Although developers often avoided updates due to the fear of leaking defects or crashes into production, the engineering team established a risk-assessment framework based on codebase usage and the volume of incoming changes. To minimize update diffs and avoid accumulating large issues, libraries are updated incrementally on a weekly cadence matching upstream release schedules. Automated UI test cases written in Gherkin syntax execute on CI for every merge request, while high-risk bumps trigger targeted QA manual testing. This structured updating process prevents defect leakage while keeping the application compliant with evolving Google Play target API requirements.

  Lucas Nelaupe

• Grab · Dec 22, 2020

  Pharos - Searching Nearby Drivers on Road Network at Scale

  Ride-hailing allocation requires identifying nearby drivers based on actual routing distance and estimated time of arrival rather than straight-line haversine distance. Grab developed Pharos, a distributed in-memory spatial microservice that evaluates road networks to match fast-moving drivers with pick-up requests. The system partitions OpenStreetMap road graphs by city and vehicle vertical while indexing driver positions and edge-based nodes using Adaptive Radix Trees. To locate candidates, Pharos projects driver coordinates onto road segments as phantom nodes and runs Incremental Network Expansion to compute isochrone reaches. Read-write concurrency is managed by switching tree root snapshots, allowing concurrent reads while updates produce new roots for subsequent lookups.

  Hao Wu

• Grab · Nov 22, 2021

  Using real-world patterns to improve matching in theory and practice

  Continuous ride-hailing assignment relies on solving the minimum weight bipartite matching problem between passengers and driver-partners. While traditional implementations assume a precalculated cost matrix, computing shortest-path travel times across large road networks dominates total execution time. Researchers introduced an Incremental Kuhn-Munkres algorithm that leverages the spatial locality of optimal matches to compute edge costs on demand. The approach integrates priority queues and lower-bounding techniques with refinement rules to avoid evaluating distant pairs while guaranteeing the same optimal assignment. Evaluated on Singapore road network data and real Grab production workloads, the incremental techniques reduced exact cost calculations and decreased assignment running times by over an order of magnitude.

  Tenindra Abeywickrama