# The complete stream processing journey on FlinkSQL

[Grab](https://yomu.fyi/company/grab) · Calvin Tran · Jun 12, 2025

## Summary

Grab previously relied on Apache Zeppelin notebooks for interactive stream processing exploration, but faced lagging Flink version upgrades, five-minute cluster cold starts, and poor integration with internal platforms. To address these limitations, the team migrated to a shared FlinkSQL gateway architecture structured into compute, integration, and query layers. The new setup uses a Hive Metastore catalog to expose Kafka topics as relational tables, while a custom control plane handles authentication and headless REST APIs over Flink's native interface. For production workflows, a configuration-based portal accepts SQL logic and automatically provisions and deploys Flink pipelines within ten minutes. This transition reduced ad-hoc query response times to under one minute and eliminated the need to maintain version adapter shims.

## Takeaways

- Replacing isolated per-user Zeppelin clusters with a shared FlinkSQL gateway reduced interactive query result latency from five minutes to one minute.
- Using FlinkSQL gateway instead of independent notebook frameworks eliminated the need to maintain custom shims across divergent Apache Flink versions.
- A custom control plane adds headless REST APIs with operation IDs for polling paginated results, allowing downstream internal platforms to query Kafka streams programmatically.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Developer Experience](https://yomu.fyi/topic/developer-experience), [Kafka](https://yomu.fyi/topic/kafka), [Streaming](https://yomu.fyi/topic/streaming)

[Read original post](https://engineering.grab.com/the-complete-stream-processing-journey-on-flinksql)