Rethinking Stream Processing: Data Exploration

Summary

Online data materialised in Kafka topics at Grab traditionally remained uninspected until ingested into the Data Lake as offline data, introducing latency and making stream processing applications difficult to develop. To enable earlier data exploration, Grab adopted Apache Zeppelin notebooks integrated with an internal data streaming platform. Zeppelin uses a Flink interpreter to convert interactive SQL queries into Flink jobs submitted to a Flink session cluster. The platform dynamically translates Protobuf schemas into SQL Data Definition Language statements to simplify table discovery. Security and compliance risks are addressed by combining mutual TLS authentication with Strimzi's Open Policy Agent to enforce read-only access policies and audit logging on Kafka topics.

Takeaways

1. 1

   Zeppelin notebooks pass interactive user queries to a Flink interpreter, which generates a JAR file and submits it to a Flink session cluster to execute and return results.

2. 2

   Integrated tooling maps the Protobuf schema definition of a Kafka stream directly to SQL DDL, eliminating the need to manually write table definitions.

3. 3

   Data access authorization and audit trails are managed by authenticating users via mTLS certificates and enforcing read-only topic access policies through Strimzi's Open Policy Agent.