# The journey of building a comprehensive attribution platform

[Grab](https://yomu.fyi/company/grab) · Kang Huang · Feb 20, 2024

**Type:** Problem & solution

## Summary

Grab needed to modernize its marketing analytics from manual ad hoc queries and high data latency to a platform supporting real-time attribution for pricing models like cost per order. The engineering team initially deployed a pure stream-processing engine using Kappa architecture, Kafka, ScyllaDB, and Redis, which reduced latency from days to minutes and merged ads and promo touchpoints. However, stream-only processing faced high costs, out-of-order event issues, and difficulties running multi-touch models across longer historical windows. Grab transitioned to a Lambda architecture pairing Coban stream processing with Spark-based batch ETL and Amazon S3. This hybrid design separated real-time operational metrics from historical batch reporting, cutting real-time processing costs by approximately 25% while maintaining under 1% data discrepancy.

## Context

Grab relied on ad hoc queries and manual data preparation that suffered from 2–3 day latencies, discrepant results, and recurring pipeline issues while needing near real-time attribution for cost per order pricing.

## Approach / What changed

Grab built a streaming Kappa attribution engine with Kafka and Druid, then evolved it into a Lambda architecture that combined an in-house Coban real-time streaming layer with Spark batch ETL jobs and Amazon S3 storage.

## Takeaways

- Kappa architecture reduced attribution data latency from 2–3 days down to a few minutes to support real-time decision-making.
- A pure streaming approach struggled with out-of-order events, multi-touch attribution complexity, and scaling costs over longer historical attribution windows.
- Transitioning to a Lambda architecture reduced real-time data processing costs by approximately 25% and brought data discrepancies below 1%.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Kafka](https://yomu.fyi/topic/kafka), [Performance](https://yomu.fyi/topic/performance), [Streaming](https://yomu.fyi/topic/streaming)

[Read original post](https://engineering.grab.com/attribution-platform)