Loading post…
GrabHuong VuongProblem & solution
Building a Spark observability product with StarRocks: Real-time and historical performance analysis
Summary
Grab’s Iris project replaces a fragmented Spark observability stack with a centralised platform for real-time and historical analysis. The previous setup split data between Grafana, Superset, InfluxDB, and a data lake, creating usability, access-control, pipeline, metadata, and high-cardinality query challenges. The new architecture uses StarRocks as a unified store, ingesting Kafka directly, serving a custom Iris web application and Superset, and backing up to S3. Its relational data model links worker metrics through worker_uuid and Spark event metrics through app_id, while materialised views and dynamic partitioning support responsive queries and data retention as volumes grow. The resulting system consolidates metrics and metadata, simplifies ingestion and offline processing, and supports job-level showback/chargeback; the roadmap includes recommendations, advanced analytics, machine learning, and scalability improvements.
Context
Iris’s original observability setup separated real-time and historical data across Grafana, Superset, InfluxDB, and a data lake. This created a fragmented user experience, limited permissions, complex offline pipelines, difficulties handling string metadata and joins, and performance problems with high-cardinality identifiers.
Approach / What changed
The redesigned Iris platform uses StarRocks for real-time and historical storage, direct Kafka ingestion, a custom web application, Superset integration, and scheduled S3 backups. Its schema links cluster metadata, worker metrics, and Spark events through worker_uuid and app_id, with materialised views and dynamic partitioning supporting query performance and retention.
Takeaways
StarRocks provides a single data store for Spark metrics and metadata, replacing separate real-time and historical systems.
Direct Kafka ingestion removes Telegraf from the pipeline, while scheduled StarRocks-to-S3 backups simplify offline data handling.
Materialised views improve response times for common UI operations, and dynamic partitioning manages retention as data volumes grow.
Source and freshness
- Source
- engineering.grab.com
- Published
- Added to Yomu
- Last updated
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.