Enabling near real-time data analytics on the data lake

Summary

Traditional data lake setups using Parquet on Hive metastores struggle with frequent updates and long pipeline intervals, introducing significant latency for ad hoc queries. Grab solved this by implementing Apache Hudi to support near real-time analytics across bounded relational databases and unbounded Kafka streams. For high-throughput sources, Flink streams Avro log files to Merge On Read tables and generates compaction plans for asynchronous Spark writers. Low-throughput workloads leverage Copy On Write tables, while relational database sources ingest binlogs via Flink Change Data Capture connectors. This architecture reduced analytics data latency to the minute level without overloading production databases and Kafka clusters.

Context

Cloud object storage with Hive metastore and Parquet files struggled with frequent data updates and multi-step scheduled downstream transformations, creating high query latencies for ad hoc analytics.

Approach / What changed

Adopted Apache Hudi with tailored ingestion pipelines: Flink stream writers write Avro log files to Merge On Read tables and plan compactions for Spark for high-throughput Kafka streams, Copy On Write tables handle low-throughput sources, and Flink CDC connectors replicate relational database binlogs.

Takeaways

• Compaction planning was offloaded to the Flink writer as an asynchronous service, allowing a single actor to orchestrate table services while separate Spark jobs execute the compaction plans.
• Partitioning unbounded Kafka data by event time down to the hour level bounded Parquet file sizes and optimized compaction planning using BoundedPartitionAwareCompactionStrategy.
• Grab selected Hudi's Bucket Index over Flink State Index to avoid unbounded in-memory state growth and deployment preservation complexities for Kafka streams.

Related reading

• Grab · Dec 1, 2022

  How KartaCam powers GrabMaps

  Grab's Geo team required an efficient, low-cost way to collect fresh street-level imagery across Southeast Asia, where professional mapping equipment is prohibitively expensive and smartphone crowdsourcing yields inconsistent quality. To address this, Grab built KartaCam, a custom mapping device equipped with a 12MP sensor, dual-band GNSS, 4G LTE, and onboard edge AI. Edge machine learning models evaluate scene suitability, check image quality, filter for map-relevant objects, and blur personal data directly on the device prior to upload. Deploying a four-camera KartaCam 360 array delivers panoramic coverage and point-of-interest data comparable to commercial mapping rigs at roughly one-twentieth the hardware cost.

  Shuangquan Hou

• Grab · Nov 8, 2023

  Graph modelling guidelines

  Graph modelling uses graph theory to represent real-world entities, relationships, and properties through nodes and edges, unlocking insights across interconnected datasets. The process begins with defining domain concepts, mapping entities and relationships, and assigning relevant descriptive properties to both nodes and edges. Practitioners choose between graph structures such as property graphs or Resource Description Framework models before optionally developing a structural schema. Data is then imported or generated to populate storage systems such as Neo4j, Amazon Neptune, Azure Cosmos DB, or in-memory data structures. Finally, the graph is visualised and analysed using algorithmic methods like shortest path calculation, centrality measurement, and community detection.

  Wenxiang Lu

• Grab · Oct 25, 2022

  Using mobile sensor data to encourage safer driving

  Dedicated vehicular inertial measurement units are prohibitively expensive to install across large fleets, while customer feedback lacks the coverage required to assess driving safety comprehensively. To monitor driver behavior at scale, Grab processes smartphone sensor data collected via a mobile telematics SDK at high sampling frequencies. A custom orientation algorithm calculates Euler roll and pitch angles from gravity and resolves yaw through principal component analysis to align phone measurements with vehicle axes. The pipeline applies low-pass or Kalman filtering to handle varying noise levels, followed by threshold peak-finding algorithms to detect harsh acceleration, braking, and cornering. Speeding events are identified by comparing GPS speeds against known map limits or aggregated speed profiles derived from free-flowing traffic.

  Laiyi Lin

• Grab · Aug 2, 2023

  Unsupervised graph anomaly detection - Catching new fraudulent behaviours

  Fraud detection systems relying on historical training data often struggle to detect new fraudulent modus operandi because emerging patterns lack labeled supervision. Grab developed GraphBEAN, an unsupervised autoencoder model designed to detect anomalous patterns across bipartite interaction graphs representing consumers and merchants. The architecture uses graph convolution layers to encode node and edge attributes, then reconstructs features and edge existence through separate feature and structure decoders. High reconstruction errors flag rare, anomalous behaviors at both the node and edge levels. An automated pipeline applies heuristic fraud tags to these scores and routes them to human analysts and automated mitigation systems.

  Rizal Fathony