# Powering Partner Gateway metrics with Apache Pinot

[Grab](https://yomu.fyi/company/grab) · Alvis Chew · Sep 23, 2025

## Summary

Grab needed to power real-time analytics dashboards for its Partner Gateway, tracking API status codes and latency across datasets reaching 6.8 billion rows over 30-day windows. Initial aggregation queries on datasets exceeding 150GB frequently timed out past 10 seconds, failing to meet the platform's 300-millisecond service level agreement. To support low-latency Online Analytical Processing queries, Grab routed metric streams through Apache Kafka and Apache Flink into Apache Pinot. Query execution was then accelerated by partitioning Kafka topics by metric name, adding rounded time interval columns, and implementing Star-tree indexes for multidimensional pre-aggregation.

## Takeaways

- To enable message deserialization via Apache Pinot's Confluent Schema Registry Protobuf Decoder plugin, Apache Flink prepends a magic byte and Schema ID to Kafka messages generated by Grab's Go services.
- Partitioning the metric Kafka topic by metric name optimizes Pinot query pruning, reduces I/O scanning overhead, and enables parallel metric processing across distributed nodes.
- Configuring a Star-tree index with defined dimension split orders and star nodes allows Pinot to resolve dimensional aggregations by scanning single pre-aggregated documents instead of multi-record datasets.

**Tags:** [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Kafka](https://yomu.fyi/topic/kafka), [Performance](https://yomu.fyi/topic/performance), [Scalability](https://yomu.fyi/topic/scalability), [Streaming](https://yomu.fyi/topic/streaming)

[Read original post](https://engineering.grab.com/pinot-partnergateway-tech-blog)