How Apache Flink Processes Streaming Data
Apache Flink is a distributed stream-processing framework that ingests continuous data, applies transformations in parallel, and outputs results with fault tolerance and low latency.
Loading post…
GrabCalvin TranProblem & solution
The complete stream processing journey on FlinkSQL
Summary
The post describes replacing per-user Zeppelin notebooks with a shared FlinkSQL gateway to improve interactive stream processing over Kafka data and support production pipelines. Zeppelin notebooks were limited by support for Flink 1.17, roughly five-minute cold starts, and isolation from internal platforms, while current Flink was 1.20. The replacement uses compute, integration, and query layers: a shared FlinkSQL gateway submits SQL through its REST API to session-cluster jobs, while a custom control plane handles authentication, authorization, session management, orchestration, and paginated results for UI and headless API clients. An interactive UI uses a Hive Metastore catalog to expose Kafka topics as tables, and the platform returns results in one minute rather than five; a configuration-based deployment tool provisions resources and deploys SQL-defined Flink pipelines within ten minutes. After release, the post reports an uptick in created pipelines and interactive queries, but it does not quantify that increase.
Context
The previous Zeppelin-based Flink solution created technical roadblocks: its latest supported Flink version was 1.17 while Flink had reached 1.20, per-user clusters took roughly five minutes to start, and the notebook setup was difficult to integrate with dashboards, automated pipelines, and other internal platforms.
Approach / What changed
The solution replaces per-user Zeppelin clusters with a shared FlinkSQL gateway organized into compute, integration, and query layers. A custom control plane adds authentication, authorization, session management, orchestration, and APIs over Flink's REST interface, while an interactive UI and a configuration-based deployment tool support SQL exploration, integration, and production pipeline deployment.
Takeaways
A shared FlinkSQL session cluster reduced the time to first displayed results from roughly five minutes with Zeppelin to one minute, although task managers are still provisioned on demand.
The custom control plane abstracts Flink's native REST API and provides authenticated sessions, orchestration, and paginated results for both the interactive UI and headless API clients.
The deployment tool lets users configure connectors, SQL logic, parallelism, and resources; it dynamically builds a Flink job graph and deploys the pipeline to production within 10 minutes.
Source and freshness
- Source
- engineering.grab.com
- Published
- Added to Yomu
- Last updated
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
Visual explainers
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.