Loading…
Data Pipelines
65 posts about Data Pipelines. Every summary links to the original.
Grab ·
Improving Hugo stability and addressing oncall challenges through automation
Grab's Hugo platform manages over 4,000 data ingestion pipelines for the company's data lake, but recurring failures caused significant on-call strain and data downtime due to complex triage processes. To resolve these operational challenges, an automated architecture was introduced featuring signal collection, automated diagnosis, root-cause tracking, auto-resolution, and a central dashboard. Rather than parsing extensive execution logs from Spark or Airflow, the diagnosis engine analyzes direct computation signals and temporal execution steps in parallel to pinpoint assignees and root causes. An asynchronous auto-resolution framework executes custom recovery handlers, such as automated backoff retries during database replica lags. This automated workflow reduced on-call workloads, improved dataset visibility, and shortened issue triage times across Grab's data ecosystem.
Shuguang XiangGrab ·
Building a Spark observability product with StarRocks: Real-time and historical performance analysis
Grab redesigned its Spark observability platform, Iris, to overcome limitations associated with its previous Telegraf, InfluxDB, and Grafana stack. InfluxDB presented operational challenges due to limited SQL compatibility, poor handling of string metadata, and query degradation on high-cardinality identifiers. The team replaced InfluxDB with StarRocks to serve as a unified analytical engine for both real-time cluster metrics and historical analysis. StarRocks ingests metrics directly from Kafka via routine load tasks, storing worker and Spark event data in partitioned duplicate-key OLAP tables linked by worker and application identifiers. This architecture eliminated intermediate ingestion agents, simplified S3 data lake backups, and enabled a custom web application alongside Superset for consistent querying.
Huong VuongGrab ·
Metasense V2: Enhancing, improving and productionisation of LLM powered data governance
Grab scaled Metasense to automate metadata generation, column-level classification, and sensitivity tiering across its entire data lake. Post-rollout analysis revealed that high tag volumes, wide tables, and mixed content—such as nested JSON and customer communications—strained model capacity and led to missed Personally Identifiable Information (PII). To overcome these limitations, the team split the classification process into distinct PII and non-PII tasks, reduced prompt word counts, and partitioned tables with over 150 columns into smaller units. Integrating LangChain and LangSmith modernized the architecture by enabling direct prompt experimentation, custom metric tracking, and version-controlled deployments. The updated pipeline achieved low misclassification rates alongside automated alert thresholds to trigger model improvement protocols if errors increase.
Nick BuhrerGrab ·
Unveiling the process: The creation of our powerful campaign builder
Grab details the event processing architecture behind Trident, its internal marketing campaign platform that evaluates If This, Then That (IFTTT) logic over Kafka streams. The core processing unit is a treatment consisting of an event, optional conditions, and actions. Complex campaign capabilities—such as counters, limits, and delays across multiple hours via recursive SQS message scheduling—are assembled from multiple coordinated treatments. To simplify campaign creation, Grab introduced a flowchart-like visual builder represented as a JSON node tree that compiles recursively into treatments while persisting node-to-treatment mappings to reconcile edits over time.
Jie ZhangGrab ·
LLM-powered data classification for data entities at scale
Grab needed to classify sensitive data at the table and column level across petabytes of database tables and streaming schemas. Manual schema-tiering campaigns had resulted in half of all schemas receiving overly strict Tier 1 access controls, while an initial automated service using regex patterns and third-party machine learning produced high false-positive rates and lacked customizability. To address this, the Caspian data engineering and governance teams enhanced their internal orchestration service, Gemini, by integrating GPT-3.5 via Azure OpenAI. Gemini aggregates classification requests into mini-batches, handles API rate limits, and uses prompt engineering—including few-shot examples, curated tag libraries, and explicit JSON DTO schemas—to reliably tag columns for data owner verification.
Hualin LiuGrab ·
How we evaluated the business impact of marketing campaigns
Grab needed a scalable, centralised way to evaluate the business impact of millions of marketing messages sent across expanding campaigns. Manual analysis by marketing analysts produced inconsistent results and failed to scale, leading to the creation of multi-touch attribution and controlled experiment methodologies. The team designed a Marketing attribution model (MAM) that credits touchpoints based on relevance and recency, while using stratified hold-out groups to assess single campaign lift against baseline metrics. Statistical significance calculations, implemented using the jStat library based on the central limit theorem, ensure observed business metric differences like gross merchandise value are not random. The system provides campaign managers with direct access to impact analyses while the team works on challenges surrounding smaller sample sizes and hold-out group constraints.
Jie ZhangGrab ·
Ensuring data reliability and observability in risk systems
Grab's in-house risk management platform, GrabDefence, relies on ingesting large volumes of upstream data to drive real-time heuristic rules and data science models. To prevent data discrepancies and missing information from degrading fraud detection, the team implemented a real-time data observability pipeline. The solution uses Apache Flink SQL alongside custom table functions like JSONEXPLOAD to flatten nested JSON structures and aggregate metrics over 5-minute tumbling windows. Processed metrics are sent to Datadog, where counters are grouped by source stream and evaluated using anomaly detection algorithms. Alerts are routed directly to Slack, reducing anomaly detection latency from days or weeks down to within the same day or hour.
Yi Ni OngGrab ·
Grab Experiment Decision Engine - a Unified Toolkit for Experimentation
Grab's Experimentation platform tests thousands of experimental variants weekly, but analyses were largely handled manually using disparate tools unsuited to multi-sided marketplace nuances. To standardize workflows and eliminate reliability issues, Grab developed the GrabX Decision Engine, an internally open-sourced Python toolkit integrated with its Automated Experiment Analysis application. The framework includes a Trusted Advisor component that automates power analysis and sample size estimation while adjusting for clustered standard errors across metric data types. It also bundles a post-experiment statistical toolbox executing default and custom evaluations, ranging from non-parametric tests to regression models that control for confounders and fixed effects. This unified platform institutionalizes experimentation best practices across teams, streamlining collaboration and ensuring consistent statistical inference.
Ruike ZhangGrab ·
Iris - Turning observations into actionable insights for enhanced decision making
Standard infrastructure monitoring tools track CPU and memory usage at the host or container level, obscuring the precise resource consumption of individual Spark jobs on shared instances. To achieve granular observability, the Iris platform gathers job metrics directly at the Java Virtual Machine level without requiring changes to user application code. Data collection relies on Uber's JVM Profiler for deep JVM, thread, and memory pool diagnostics alongside sparkMeasure for stage and task execution statistics. Collected metrics route asynchronously through an Apache Kafka queue to avoid execution bottlenecks, feeding a Telegraf, InfluxDB, and Grafana stack for real-time monitoring while archiving to a data lake for offline analytics. This multi-tiered telemetry enables engineering teams to identify over- or under-provisioned jobs and optimize ETL resource allocations.
Huong VuongGrab ·
Enabling near real-time data analytics on the data lake
Traditional data lake setups using Parquet on Hive metastores struggle with frequent updates and long pipeline intervals, introducing significant latency for ad hoc queries. Grab solved this by implementing Apache Hudi to support near real-time analytics across bounded relational databases and unbounded Kafka streams. For high-throughput sources, Flink streams Avro log files to Merge On Read tables and generates compaction plans for asynchronous Spark writers. Low-throughput workloads leverage Copy On Write tables, while relational database sources ingest binlogs via Flink Change Data Capture connectors. This architecture reduced analytics data latency to the minute level without overloading production databases and Kafka clusters.
Shi Kai NgGrab ·
The journey of building a comprehensive attribution platform
Grab needed to modernize its marketing analytics from manual ad hoc queries and high data latency to a platform supporting real-time attribution for pricing models like cost per order. The engineering team initially deployed a pure stream-processing engine using Kappa architecture, Kafka, ScyllaDB, and Redis, which reduced latency from days to minutes and merged ads and promo touchpoints. However, stream-only processing faced high costs, out-of-order event issues, and difficulties running multi-touch models across longer historical windows. Grab transitioned to a Lambda architecture pairing Coban stream processing with Spark-based batch ETL and Amazon S3. This hybrid design separated real-time operational metrics from historical batch reporting, cutting real-time processing costs by approximately 25% while maintaining under 1% data discrepancy.
Kang HuangGrab ·
Rethinking Stream Processing: Data Exploration
Online data materialised in Kafka topics at Grab traditionally remained uninspected until ingested into the Data Lake as offline data, introducing latency and making stream processing applications difficult to develop. To enable earlier data exploration, Grab adopted Apache Zeppelin notebooks integrated with an internal data streaming platform. Zeppelin uses a Flink interpreter to convert interactive SQL queries into Flink jobs submitted to a Flink session cluster. The platform dynamically translates Protobuf schemas into SQL Data Definition Language statements to simplify table discovery. Security and compliance risks are addressed by combining mutual TLS authentication with Strimzi's Open Policy Agent to enforce read-only access policies and audit logging on Kafka topics.
Shi Kai NgGrab ·
Road localisation in GrabMaps
Grab needed to localise nearly 30 million road segments across more than 10,000 area boundaries in Southeast Asia to attach hyperlocal attributes like language, driving side, and vehicle access rules. Testing direct geometric inclusion between complex road polylines and boundary polygons proved computationally prohibitive for daily map generation pipelines. To optimize this process, Grab introduced geohashes as rectangular proxies, precomputing geohash coverage across borders and road segments before joining the datasets in parallel. To resolve misclassification errors near borders without sacrificing performance, the team evaluated geohash coverage percentages and refined boundary-touching geohashes with polygon clipping. A laptop benchmark on a subset of data reduced runtime from 38 minutes with geometric intersection to 78 seconds using the geohash approximation.
Roxana CrisanGrab ·
Scaling marketing for merchants with targeted and intelligent promos
Grab previously relied on globally assigned, heuristic promotional campaigns where all users could redeem offers until limits were reached. This lack of targeting and customisation failed to optimize promo spending or meet specific merchant business objectives. To solve this, the engineering team built Bullseye, an automated AI-driven promotional assignment system that customises and assigns offers to consumers. The architecture integrates an operations user interface, backend services, Amazon S3 storage, and Spark jobs executing a suite of predictive modules including eater segmentation, campaign impact simulation, and customer response modeling. Since deploying Bullseye in 2021, the system has increased food campaign sales while decreasing promo spend across food and Mart campaigns.
Sharon TengGrab ·
Building hyperlocal GrabMaps
Southeast Asia presents unique mapping challenges, including fast-changing road restrictions, new roads appearing rapidly, and severe traffic congestion. To address these conditions, GrabMaps integrates hyperlocal data derived from local team knowledge, rides and deliveries GPS pings, and KartaView imagery with OpenStreetMap data. The pipeline downloads OpenStreetMap .pbf files for Asia and converts them into Parquet format using a Parquetizer. GrabMaps processes spatial data using Apache Spark and bi-directional graphs divided by geohash to detect complex features such as intersection links based on road parallelism, naming, direction, and shape geometry. Tagging these intersection links enables downstream navigation services like GrabNav to calculate shorter routing alternatives and provide more accurate pricing.
Adriana LazarGrab ·
Unsupervised graph anomaly detection - Catching new fraudulent behaviours
Fraud detection systems relying on historical training data often struggle to detect new fraudulent modus operandi because emerging patterns lack labeled supervision. Grab developed GraphBEAN, an unsupervised autoencoder model designed to detect anomalous patterns across bipartite interaction graphs representing consumers and merchants. The architecture uses graph convolution layers to encode node and edge attributes, then reconstructs features and edge existence through separate feature and structure decoders. High reconstruction errors flag rare, anomalous behaviors at both the node and edge levels. An automated pipeline applies heuristic fraud tags to these scores and routes them to human analysts and automated mitigation systems.
Rizal FathonyGrab ·
PII masking for privacy-grade machine learning
Data engineers at Grab require real-world streaming data to tune machine learning models, but user personal data cannot be exposed in non-production environments. To solve this, the data streaming team implemented an automated masking pipeline between production and staging Kafka clusters. Developers explicitly tag Personally Identifiable Information (PII) types in Protocol Buffers schemas, which a CI script validates against a keyword list before packaging the schemas into Scala JARs. An in-house Apache Flink application running in production consumes the unmasked streams, applies dynamic and consistent transformations like keyed HMACs or seeded random values, and produces sanitised records to staging. Untagged new fields in schema updates are automatically dropped by the deserialiser until the masking application is redeployed, preventing accidental data leaks.
Fabrice HarbulotGrab ·
Graph service platform
Grab's GrabDefence team required a dedicated graph infrastructure to proactively identify mobile fraud patterns, such as multiple accounts operating on shared physical devices and suspicious financial loops. To address this, the team built a four-layer Platform as a Service that encapsulates graph database operations behind uniform RESTful APIs for OLTP search and OLAP analysis. The architecture utilizes Amazon S3 for raw data files, Amazon Neptune for graph storage, DynamoDB for schema and metadata configurations, and Kafka for streaming ingestion. When users trigger data loading tasks, the service validates entity attributes against schemas stored in DynamoDB before importing records into Neptune. This infrastructure allows investigators to traverse adjacent account IDs and visualize complex entity relationships without managing underlying database runtimes.
Wenxiang LuGrab ·
Query expansion based on user behaviour
Grab developed a query expansion framework to resolve search issues such as typos, multi-language terms, missing alternatives, and word spacing. The corpus is generated through manual annotation for the top thousand fat-head queries and session-based data mining of user rewrite patterns. Offline pipelines extract adjacent query rewrites occurring within 30 seconds where only the second query generated a click, limiting candidates to three per query for serving efficiency. In production, expanded keywords pass through a query understanding, multi-recall, and result fusion flow with automated pruning of underperforming pairs. Online A/B testing across six Southeast Asian countries showed improvements in click-through and conversion rates, with data-mined expansions outperforming manual annotations alone.
Shuailong LiangGrab ·
Using mobile sensor data to encourage safer driving
Dedicated vehicular inertial measurement units are prohibitively expensive to install across large fleets, while customer feedback lacks the coverage required to assess driving safety comprehensively. To monitor driver behavior at scale, Grab processes smartphone sensor data collected via a mobile telematics SDK at high sampling frequencies. A custom orientation algorithm calculates Euler roll and pitch angles from gravity and resolves yaw through principal component analysis to align phone measurements with vehicle axes. The pipeline applies low-pass or Kalman filtering to handle varying noise levels, followed by threshold peak-finding algorithms to detect harsh acceleration, braking, and cornering. Speeding events are identified by comparing GPS speeds against known map limits or aggregated speed profiles derived from free-flowing traffic.
Laiyi Lin