# Rethinking Stream Processing: Data Exploration

[Grab](https://yomu.fyi/company/grab) · Shi Kai Ng · Jan 31, 2024

**Type:** Problem & solution

## Summary

Online data materialised in Kafka topics at Grab traditionally remained uninspected until ingested into the Data Lake as offline data, introducing latency and making stream processing applications difficult to develop. To enable earlier data exploration, Grab adopted Apache Zeppelin notebooks integrated with an internal data streaming platform. Zeppelin uses a Flink interpreter to convert interactive SQL queries into Flink jobs submitted to a Flink session cluster. The platform dynamically translates Protobuf schemas into SQL Data Definition Language statements to simplify table discovery. Security and compliance risks are addressed by combining mutual TLS authentication with Strimzi's Open Policy Agent to enforce read-only access policies and audit logging on Kafka topics.

## Context

Online streaming data in Kafka topics was underutilised and difficult to explore prior to Data Lake ingestion, hindering data end users from quickly understanding schemas, testing logic, and building stream processing applications.

## Approach / What changed

Grab integrated Apache Zeppelin notebooks supporting Flink SQL into its streaming platform control plane, automated Protobuf-to-DDL derivation, and secured Kafka topic access using mTLS and Strimzi Open Policy Agent integration.

## Takeaways

- Zeppelin notebooks pass interactive user queries to a Flink interpreter, which generates a JAR file and submits it to a Flink session cluster to execute and return results.
- Integrated tooling maps the Protobuf schema definition of a Kafka stream directly to SQL DDL, eliminating the need to manually write table definitions.
- Data access authorization and audit trails are managed by authenticating users via mTLS certificates and enforcing read-only topic access policies through Strimzi's Open Policy Agent.

**Tags:** [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Developer Experience](https://yomu.fyi/topic/developer-experience), [Kafka](https://yomu.fyi/topic/kafka), [Kubernetes](https://yomu.fyi/topic/kubernetes), [Streaming](https://yomu.fyi/topic/streaming)

[Read original post](https://engineering.grab.com/rethinking-streaming-processing-data-exploration)