The complete stream processing journey on FlinkSQL

Summary

Grab previously relied on Apache Zeppelin notebooks for interactive stream processing exploration, but faced lagging Flink version upgrades, five-minute cluster cold starts, and poor integration with internal platforms. To address these limitations, the team migrated to a shared FlinkSQL gateway architecture structured into compute, integration, and query layers. The new setup uses a Hive Metastore catalog to expose Kafka topics as relational tables, while a custom control plane handles authentication and headless REST APIs over Flink's native interface. For production workflows, a configuration-based portal accepts SQL logic and automatically provisions and deploys Flink pipelines within ten minutes. This transition reduced ad-hoc query response times to under one minute and eliminated the need to maintain version adapter shims.

Takeaways

• Replacing isolated per-user Zeppelin clusters with a shared FlinkSQL gateway reduced interactive query result latency from five minutes to one minute.
• Using FlinkSQL gateway instead of independent notebook frameworks eliminated the need to maintain custom shims across divergent Apache Flink versions.
• A custom control plane adds headless REST APIs with operation IDs for polling paginated results, allowing downstream internal platforms to query Kafka streams programmatically.

Related reading

• Grab · Sep 23, 2025

  Powering Partner Gateway metrics with Apache Pinot

  Grab needed to power real-time analytics dashboards for its Partner Gateway, tracking API status codes and latency across datasets reaching 6.8 billion rows over 30-day windows. Initial aggregation queries on datasets exceeding 150GB frequently timed out past 10 seconds, failing to meet the platform's 300-millisecond service level agreement. To support low-latency Online Analytical Processing queries, Grab routed metric streams through Apache Kafka and Apache Flink into Apache Pinot. Query execution was then accelerated by partitioning Kafka topics by metric name, adding rounded time interval columns, and implementing Star-tree indexes for multidimensional pre-aggregation.

  Alvis Chew

• Grab · Jul 7, 2025

  DispatchGym: Grab’s reinforcement learning research framework

  Applying reinforcement learning to dispatch systems is often hindered when the chosen control levers exert weak influence over reward functions. To streamline research, Grab built DispatchGym, a framework that connects reinforcement learning algorithms to a dispatch process simulation via the Gymnasium API. The simulation emphasizes directional accuracy over absolute precision, allowing researchers to evaluate relative metric shifts across supply and demand scenarios. Built in modular Python and accelerated with Numba, the system allows data scientists to test code locally and launch distributed Spark executions with a single command-line call. The framework has been used to evaluate various contextual bandit models and action sampling strategies for tuning dispatch hyperparameters.

  Tan Sien Yi

• Grab · Jan 29, 2026

  Cursor at Grab: Adoption and impact

  Following a multi-tool AI strategy, Grab integrated the AI coding assistant Cursor into its engineering toolkit in late 2024 to accelerate software development. Technical staff adoption reached 98% monthly active usage with a 50% suggestion acceptance rate, supported by custom monorepo indexing and preconfigured rules aligned with internal coding conventions. Engineers frequently apply the tool to unit test generation, code refactoring, cross-repository navigation, and routine API scaffolding, with over a third of merge requests incorporating Cursor. The rollout also encompasses non-technical personnel and product designers who, after receiving Git training, submit direct production UI fixes. Statistical evaluations using fixed-effects regression indicate a dose-response relationship between Cursor usage intensity and measurable productivity gains.

  Akshay Misra

• Grab · Oct 6, 2025

  Highly concurrent in-memory counter in GoLang

  High database CPU utilization caused by relentless write traffic for marketing campaign counts prompted Grab to build an in-memory aggregation buffer rather than migrate from SQL to NoSQL. The team implemented an in-memory counter in Go that batches usage updates and periodically flushes them to persistent storage. To eliminate the serialization bottlenecks of mutex-locked maps under high concurrency, the design adopted Go's sync.Map using atomic CompareAndSwap retries for value updates alongside LoadAndDelete for periodic background flushes. Because a finite set of campaign keys is repeatedly accessed, operations hit the internal read map fast path nearly 99% of the time, achieving a threefold throughput improvement over standard mutex locks in benchmarks. In production, this architecture decreased database update queries by 68% and reduced master database CPU utilization from 35% to 18%.

  Naveen Kumar Jakuva Premkumar