# Plumbing At Scale

[Grab](https://yomu.fyi/company/grab) · Karan Kamath · Jan 6, 2020

**Type:** Problem & solution

## Summary

Grab's backend services process terabytes of data ingress per hour, generating recurring needs for stream transformations, joins, and time-windowed aggregations across diverse workloads. To support these asynchronous processing patterns across their Go ecosystem, the Coban team developed a managed, NoOps event sourcing and stream processing platform. The architecture packages stateless processing pipelines as Kubernetes deployments on AWS, polling Kafka event logs and using ScyllaDB as a shared metastore for stateful needs like deduplication and windowing. Stream processing pods combine ingestion triggers, a worker pool runtime, and user-provided domain logic plugins with customizable failure handling. This infrastructure scales to handle over 300 billion events weekly while maintaining workload isolation and elastic autoscaling.

## Context

Grab needed a managed, NoOps platform to handle terabytes of hourly data ingress across asynchronous backend services, addressing recurring patterns such as filtering, mapping, and time-windowed event aggregations.

## Approach / What changed

The Coban team built an event sourcing and stream processing framework in Go, deploying stateless pipeline pods as Kubernetes deployments on AWS backed by Kafka event logs and ScyllaDB metastores.

## Takeaways

- Each stream processing pod separates concerns into a trigger interface for data ingestion, a runtime managing worker pools and lifecycle events, and a user-defined pipeline plugin holding domain logic.
- Pipeline deployments remain stateless by delegating intermediate storage for deduplication and time-windowed aggregations to ScyllaDB clusters, which are surfaced to backend services through GrabStats.
- Traffic elasticity and node scaling are automated in Kubernetes using the Horizontal Pod Autoscaler for pod counts and the Cluster Autoscaler for underlying worker nodes.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Go](https://yomu.fyi/topic/go), [Kafka](https://yomu.fyi/topic/kafka), [Kubernetes](https://yomu.fyi/topic/kubernetes), [Streaming](https://yomu.fyi/topic/streaming)

[Read original post](https://engineering.grab.com/plumbing-at-scale)