Rethinking Stream Processing: Data Exploration

Summary

Online data materialised in Kafka topics at Grab traditionally remained uninspected until ingested into the Data Lake as offline data, introducing latency and making stream processing applications difficult to develop. To enable earlier data exploration, Grab adopted Apache Zeppelin notebooks integrated with an internal data streaming platform. Zeppelin uses a Flink interpreter to convert interactive SQL queries into Flink jobs submitted to a Flink session cluster. The platform dynamically translates Protobuf schemas into SQL Data Definition Language statements to simplify table discovery. Security and compliance risks are addressed by combining mutual TLS authentication with Strimzi's Open Policy Agent to enforce read-only access policies and audit logging on Kafka topics.

Context

Online streaming data in Kafka topics was underutilised and difficult to explore prior to Data Lake ingestion, hindering data end users from quickly understanding schemas, testing logic, and building stream processing applications.

Approach / What changed

Grab integrated Apache Zeppelin notebooks supporting Flink SQL into its streaming platform control plane, automated Protobuf-to-DDL derivation, and secured Kafka topic access using mTLS and Strimzi Open Policy Agent integration.

Takeaways

• Zeppelin notebooks pass interactive user queries to a Flink interpreter, which generates a JAR file and submits it to a Flink session cluster to execute and return results.
• Integrated tooling maps the Protobuf schema definition of a Kafka stream directly to SQL DDL, eliminating the need to manually write table definitions.
• Data access authorization and audit trails are managed by authenticating users via mTLS certificates and enforcing read-only topic access policies through Strimzi's Open Policy Agent.

Related reading

• Grab · Jul 15, 2024

  LLM-powered data classification for data entities at scale

  Grab needed to classify sensitive data at the table and column level across petabytes of database tables and streaming schemas. Manual schema-tiering campaigns had resulted in half of all schemas receiving overly strict Tier 1 access controls, while an initial automated service using regex patterns and third-party machine learning produced high false-positive rates and lacked customizability. To address this, the Caspian data engineering and governance teams enhanced their internal orchestration service, Gemini, by integrating GPT-3.5 via Azure OpenAI. Gemini aggregates classification requests into mini-batches, handles API rate limits, and uses prompt engineering—including few-shot examples, curated tag libraries, and explicit JSON DTO schemas—to reliably tag columns for data owner verification.

  Hualin Liu

• Grab · Aug 15, 2022

  How we store and process millions of orders daily

  The Grab Order Platform processes millions of food and mart transactions daily, requiring high throughput, fault tolerance, and reduced cloud costs across transactional and analytical workloads. To meet these demands, the engineering team decoupled their database architecture by using Amazon DynamoDB for critical OLTP queries and MySQL RDS for historical OLAP queries. DynamoDB handles online order lifecycles with strong consistency, utilizing sparse Global Secondary Indexes for ongoing orders and TTL configurations to limit storage growth. Updates propagate asynchronously to MySQL RDS through a Kafka ingestion pipeline backed by Amazon SQS retries and timestamp-based version checks. This dual-database approach isolated core transaction availability from analytical queries and delivered significant cloud cost savings.

  Xi Chen

• Grab · Aug 30, 2023

  Building hyperlocal GrabMaps

  Southeast Asia presents unique mapping challenges, including fast-changing road restrictions, new roads appearing rapidly, and severe traffic congestion. To address these conditions, GrabMaps integrates hyperlocal data derived from local team knowledge, rides and deliveries GPS pings, and KartaView imagery with OpenStreetMap data. The pipeline downloads OpenStreetMap .pbf files for Asia and converts them into Parquet format using a Parquetizer. GrabMaps processes spatial data using Apache Spark and bi-directional graphs divided by geohash to detect complex features such as intersection links based on road parallelism, naming, direction, and shape geometry. Tagging these intersection links enables downstream navigation services like GrabNav to calculate shorter routing alternatives and provide more accurate pricing.

  Adriana Lazar

• Grab · Apr 9, 2024

  Grab Experiment Decision Engine - a Unified Toolkit for Experimentation

  Grab's Experimentation platform tests thousands of experimental variants weekly, but analyses were largely handled manually using disparate tools unsuited to multi-sided marketplace nuances. To standardize workflows and eliminate reliability issues, Grab developed the GrabX Decision Engine, an internally open-sourced Python toolkit integrated with its Automated Experiment Analysis application. The framework includes a Trusted Advisor component that automates power analysis and sample size estimation while adjusting for clustered standard errors across metric data types. It also bundles a post-experiment statistical toolbox executing default and custom evaluations, ranging from non-parametric tests to regression models that control for confounders and fixed effects. This unified platform institutionalizes experimentation best practices across teams, streamlining collaboration and ensuring consistent statistical inference.

  Ruike Zhang