Building a Spark observability product with StarRocks: Real-time and historical performance analysis

Summary

Grab redesigned its Spark observability platform, Iris, to overcome limitations associated with its previous Telegraf, InfluxDB, and Grafana stack. InfluxDB presented operational challenges due to limited SQL compatibility, poor handling of string metadata, and query degradation on high-cardinality identifiers. The team replaced InfluxDB with StarRocks to serve as a unified analytical engine for both real-time cluster metrics and historical analysis. StarRocks ingests metrics directly from Kafka via routine load tasks, storing worker and Spark event data in partitioned duplicate-key OLAP tables linked by worker and application identifiers. This architecture eliminated intermediate ingestion agents, simplified S3 data lake backups, and enabled a custom web application alongside Superset for consistent querying.

Takeaways

• Replacing InfluxDB with StarRocks removed the need for Telegraf by using native routine load tasks that ingest JSON-formatted metrics directly from Kafka partitions.
• The Iris data model correlates 5-second worker metrics and stage-level execution events with cluster metadata using worker_uuid and app_id fields across partitioned OLAP tables.
• StarRocks resolved InfluxDB limitations around string metadata joins, Flux query constraints, and performance degradation on high-cardinality unique identifiers.

Related reading

• Grab · Nov 26, 2025

  Real-time data quality monitoring: Kafka stream contracts with syntactic and semantic test

  Kafka streams often suffer from syntactic and semantic data quality issues that propagate undetected to downstream consumers without real-time validation. Grab addressed this challenge by developing a standardized contract testing and observability framework within its Coban platform. Stakeholders define schema rules and field-level semantic validations, which can be recommended using large language models and anonymized sample data. A transformation engine converts these contracts into inverse SQL queries executed continuously by a FlinkSQL Test Runner on a dedicated consumer group. Problematic records are published to an alert topic, archived to AWS S3, and surfaced via Slack notifications and UI field-highlighting across more than 100 critical Kafka topics.

  Yuanzhe Liu

• Grab · Jul 16, 2025

  Grab's service mesh evolution: From Consul to Istio

  Grab operated over 1,000 microservices across hybrid infrastructure using Consul alongside a fallback mechanism called Catcher. Single-point-of-failure vulnerabilities in Consul servers and limited support for multi-cluster operations prompted an evaluation of alternative mesh technologies, ultimately leading to the selection of Istio. Grab avoided the standard single-control-plane-per-cluster pattern by deploying multiple external control planes in dedicated Kubernetes clusters arranged in active-active pairs. Migration began in Q4 2024, shifting traffic across AWS and GCP while handling both HTTP and gRPC protocols with gradual traffic-shifting and rollback mechanisms.

  Hilman Kurniawan

• Grab · Oct 21, 2025

  Modernising Grab’s model serving platform with NVIDIA Triton Inference Server

  Grab's machine learning serving platform, Catwalk, experienced mounting technical debt, elevated latency, and rising costs from maintaining disparate inference engines for various frameworks. To address these limitations, the team adopted NVIDIA Triton Inference Server to establish a unified engine, starting with an ONNX migration. They built a custom Triton manager component featuring a proxy layer to translate legacy API requests alongside a server manager that handles model downloads, verification, configuration, and health checks. Within ten days, over half of online deployments migrated seamlessly without requiring client code modifications. The transition reduced p90 latency from 120ms to 20ms on large transformer models and cut average infrastructure spend by approximately 20 percent across evaluated services.

  Daniel Tai

• Grab · Nov 6, 2025

  Grab's Mac Cloud Exit supercharges macOS CI/CD

  Scaling iOS CI/CD workloads across multiple large mobile applications led to substantial expenses and queue delays under rented cloud macOS infrastructure. To mitigate network latency to Git servers and cut compute overhead, Grab relocated its build fleet from a US cloud provider to a colocation data center in Malaysia. The infrastructure deployment relies on over 200 bare-metal Mac minis across 42RU racks, purposefully avoiding virtualization solutions that introduced performance and stability penalties. Phased migration and guardrail monitoring verified cluster stability while achieving 20% to 40% reductions in pipeline and app build times. Operating the regional bare-metal cluster over an estimated three-year hardware replacement lifecycle is projected to yield 2.4 million USD in total cost of ownership savings.

  Madushan Gamage