Plumbing At Scale

Summary

Grab's backend services process terabytes of data ingress per hour, generating recurring needs for stream transformations, joins, and time-windowed aggregations across diverse workloads. To support these asynchronous processing patterns across their Go ecosystem, the Coban team developed a managed, NoOps event sourcing and stream processing platform. The architecture packages stateless processing pipelines as Kubernetes deployments on AWS, polling Kafka event logs and using ScyllaDB as a shared metastore for stateful needs like deduplication and windowing. Stream processing pods combine ingestion triggers, a worker pool runtime, and user-provided domain logic plugins with customizable failure handling. This infrastructure scales to handle over 300 billion events weekly while maintaining workload isolation and elastic autoscaling.

Context

Grab needed a managed, NoOps platform to handle terabytes of hourly data ingress across asynchronous backend services, addressing recurring patterns such as filtering, mapping, and time-windowed event aggregations.

Approach / What changed

The Coban team built an event sourcing and stream processing framework in Go, deploying stateless pipeline pods as Kubernetes deployments on AWS backed by Kafka event logs and ScyllaDB metastores.

Takeaways

• Each stream processing pod separates concerns into a trigger interface for data ingestion, a runtime managing worker pools and lifecycle events, and a user-defined pipeline plugin holding domain logic.
• Pipeline deployments remain stateless by delegating intermediate storage for deduplication and time-windowed aggregations to ScyllaDB clusters, which are surfaced to backend services through GrabStats.
• Traffic elasticity and node scaling are automated in Kubernetes using the Horizontal Pod Autoscaler for pod counts and the Cluster Autoscaler for underlying worker nodes.

Related reading

• Grab · Jan 13, 2021

  Trident - Real-time Event Processing at Scale

  Trident serves as Grab's internal real-time event-processing and workflow automation engine, driving user campaigns, rewards, and notifications across multiple business lines. To handle peak loads exceeding 2,000 events per second without duplicate execution, the system consumes decoupled Kafka streams and enforces exactly-once semantics using Redis and MySQL deduplication checks. Processing efficiency relies on server autoscaling aligned with Kafka partition counts, combined with dynamic goroutine allocation per consumer. To minimize rule evaluation overhead, Trident indexes active campaigns into an in-memory hash map by event type, cutting processing time by at least 90%. Furthermore, condition evaluation is optimized through lazy loading and a weighted sorting algorithm that checks low-cost in-memory data prior to executing expensive database queries or external service calls.

  Jie Zhang

• Grab · May 24, 2021

  Building a Hyper Self-Service, Distributed Tracing and Feedback System for Rule & Machine Learning (ML) Predictions

  Grab's Trust, Identity, Safety, and Security team processes billions of daily rule and machine learning decisions for fraud detection, safety, and identity checks. Earlier logging approaches using plain text Kibana logs and the ActionTrace library lacked structured formats, dynamic entity customization, and fine-grained access controls. To resolve these limitations, the team built Archivist, a centralized tracing, statistics, and feedback system. Archivist ingests events through an SDK into Kafka streams, buffers and routes data into Elasticsearch indices and Amazon S3, and provides a role-based user portal. The platform handles 80 million daily logs across roughly 50 business scenarios, reducing scenario onboarding times from days to minutes.

  Warren Zhou

• Grab · Jul 26, 2021

  Protecting Personal Data in Grab's Imagery

  Grab's KartaView platform collects geotagged street imagery across over 100 countries, requiring automated obfuscation of faces and licence plates to protect personal privacy. Because off-the-shelf solutions struggled with diverse global environments and equirectangular 360-degree camera formats, Grab built a custom machine learning pipeline. The system projects varied image formats into standardized planar views, applies a YOLOv4 object detection model to locate target regions, and transforms bounding coordinates back to the original imagery for blurring. Training the detector required iterative dataset updates to accommodate edge cases like face masks and mirror reflections, paired with offline view splitting and oversampling of scarce large bounding boxes. Assessments confirmed that obfuscating these regions had minimal negative impact on downstream map feature extraction services.

  Adrian Popovici

• Grab · Mar 24, 2022

  How telematics helps Grab to improve safety

  Grab developed an in-house telematics engine to monitor driver-partner habits, detect crashes, and improve ride safety beyond passenger feedback. The mobile telematics SDK collects accelerometer, gyroscope, and GPS readings, running on-device processing for immediate accident detection while batching ride data for post-trip safety reports. Because triaxial sensor sampling rates vary independently across diverse devices, the SDK synchronizes data through interpolation to a uniform time grid followed by decimation to an output data rate. Telematics signals are transformed to the frequency domain using Fourier Transform for local compression before backend upload. Grab also combines telemetry with spatial data to identify dangerous road zones and plans to expand on-device event detection across all platform verticals.

  Wilson Burhan