GrabAlvis ChewProblem & solution
Partner Gateway needed low-latency analytics for API traffic and response-time metrics across intervals from one hour to 30-day retrospectives, with a particular dataset potentially reaching approximately 6.8 billion rows in a 30-day period. Apache Pinot was selected as a distributed real-time OLAP datastore, using high-throughput ingestion and columnar, indexed, partitioned data for millisecond analytical queries. Initial aggregation queries on datasets over 150GB timed out for 20- to 30-day periods, exceeding 10 seconds and threatening a 300-millisecond SLA. The implementation routes metrics through Kafka and Apache Flink, then ingests Schema Registry-compatible messages into Pinot, while partitioning by metric name and adding aggregation-focused optimizations. For one metric, query latency fell from 5,072 to 35 milliseconds; Star-tree indexing improved speed but raised 30-day storage from 192GB to 373GB, with the additional AWS EBS cost described as minor.
Partner Gateway required real-time API performance metrics and low-latency analytical queries over intervals ranging from one-hour snapshots to 30-day retrospectives. Aggregation queries over datasets larger than 150GB frequently exceeded 10 seconds for 20- to 30-day periods, putting the 300-millisecond query SLA at risk.
The implementation uses Apache Pinot for real-time OLAP, with metrics routed through Kafka, transformed by Apache Flink, and serialized for ingestion using Protobuf and Confluent Schema Registry-compatible metadata. The table is optimized through partitioning by metric name, aggregation-interval columns, and Star-tree indexes.
Metrics include client ID, partner ID, status code, endpoint, metric name, timestamp, and value; they are uploaded to Datadog before being converted into Kafka messages.
Partitioning Kafka data by metric name improves data locality and query pruning, while allowing relevant metrics to be processed in parallel across Pinot nodes.
Star-tree indexing increased 30-day storage from approximately 192GB to 373GB, while the extra AWS gp3 EBS cost was estimated at $14.48 per month.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
Apache Pinot is a distributed real-time OLAP datastore for low-latency analytics. This architecture view shows how streams, processing, partitioning, indexed storage, and replica-aware serving fit together.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
