# Enabling near real-time data analytics on the data lake

[Grab](https://yomu.fyi/company/grab) · Shi Kai Ng · Feb 23, 2024

**Type:** Problem & solution

## Summary

Traditional data lake setups using Parquet on Hive metastores struggle with frequent updates and long pipeline intervals, introducing significant latency for ad hoc queries. Grab solved this by implementing Apache Hudi to support near real-time analytics across bounded relational databases and unbounded Kafka streams. For high-throughput sources, Flink streams Avro log files to Merge On Read tables and generates compaction plans for asynchronous Spark writers. Low-throughput workloads leverage Copy On Write tables, while relational database sources ingest binlogs via Flink Change Data Capture connectors. This architecture reduced analytics data latency to the minute level without overloading production databases and Kafka clusters.

## Context

Cloud object storage with Hive metastore and Parquet files struggled with frequent data updates and multi-step scheduled downstream transformations, creating high query latencies for ad hoc analytics.

## Approach / What changed

Adopted Apache Hudi with tailored ingestion pipelines: Flink stream writers write Avro log files to Merge On Read tables and plan compactions for Spark for high-throughput Kafka streams, Copy On Write tables handle low-throughput sources, and Flink CDC connectors replicate relational database binlogs.

## Takeaways

- Compaction planning was offloaded to the Flink writer as an asynchronous service, allowing a single actor to orchestrate table services while separate Spark jobs execute the compaction plans.
- Partitioning unbounded Kafka data by event time down to the hour level bounded Parquet file sizes and optimized compaction planning using BoundedPartitionAwareCompactionStrategy.
- Grab selected Hudi's Bucket Index over Flink State Index to avoid unbounded in-memory state growth and deployment preservation complexities for Kafka streams.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Kafka](https://yomu.fyi/topic/kafka), [Performance](https://yomu.fyi/topic/performance), [Streaming](https://yomu.fyi/topic/streaming)

[Read original post](https://engineering.grab.com/enabling-near-realtime-data-analytics)