# Iris - Turning observations into actionable insights for enhanced decision making

[Grab](https://yomu.fyi/company/grab) · Huong Vuong · Apr 3, 2024

**Type:** Problem & solution

## Summary

Standard infrastructure monitoring tools track CPU and memory usage at the host or container level, obscuring the precise resource consumption of individual Spark jobs on shared instances. To achieve granular observability, the Iris platform gathers job metrics directly at the Java Virtual Machine level without requiring changes to user application code. Data collection relies on Uber's JVM Profiler for deep JVM, thread, and memory pool diagnostics alongside sparkMeasure for stage and task execution statistics. Collected metrics route asynchronously through an Apache Kafka queue to avoid execution bottlenecks, feeding a Telegraf, InfluxDB, and Grafana stack for real-time monitoring while archiving to a data lake for offline analytics. This multi-tiered telemetry enables engineering teams to identify over- or under-provisioned jobs and optimize ETL resource allocations.

## Context

Existing monitoring tools only provided CPU and memory data at the instance or container level, bundling operating system and background tasks together and obscuring exact resource consumption for individual Spark ETL jobs on shared compute instances.

## Approach / What changed

Building Iris, an observability platform that pairs Uber's JVM Profiler and sparkMeasure within an unobtrusive Metrics Collector, funneling telemetry through a Kafka message queue into a Telegraf, InfluxDB, and Grafana (TIG) stack and a data lake.

## Takeaways

- Instance-level resource monitoring on EC2 units or Kubernetes pods mixes operating system overhead with workload metrics, preventing accurate attribution on shared compute clusters.
- Uber's JVM Profiler captures granular runtime diagnostics including buffer pools, memory pools, GC times, thread counts, and arbitrary Java method durations.
- Decoupling telemetry collection from real-time visualization and data lake storage via an asynchronous Kafka queue prevents the observability layer from creating bottlenecks in ETL workloads.

**Tags:** [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Kafka](https://yomu.fyi/topic/kafka), [Monitoring](https://yomu.fyi/topic/monitoring), [Observability](https://yomu.fyi/topic/observability), [Performance](https://yomu.fyi/topic/performance)

[Read original post](https://engineering.grab.com/iris)