# Supporting large campaigns at scale

[Grab](https://yomu.fyi/company/grab) · Jie Zhang · Apr 1, 2022

**Type:** Problem & solution

## Summary

Grab developed a batch job service within its Trident automation engine to execute multi-step marketing campaigns for millions of users simultaneously. The system replaces sequential, single-server execution with a distributed architecture powered by Apache Kafka, which distributes batches of 100 users across server clusters using hashed partition keys. To reduce network overhead and queries per second, downstream reward and messaging services introduced batch endpoints backed by bulk database queries, decreasing API latency by up to 85%. Grab further optimized performance by sharding Kafka topics by country and action type to prevent long-running reward tasks from blocking time-sensitive messaging workloads. Additionally, making terminal messaging calls asynchronous allows subsequent batch processing to proceed without waiting for message delivery confirmations.

## Context

Grab's marketing team manually triggered midnight campaigns by uploading files, leading to long execution times due to single-threaded, sequential user processing, high network latency across microservices, and idle servers in the cluster.

## Approach / What changed

Developing a batch job service within the Trident automation engine that distributes user batches across consumer servers via Kafka, builds batch endpoints with bulk database queries in downstream services, shards topics by action type and country, and executes terminal messaging calls asynchronously.

## Takeaways

- Grouping user actions into batches of approximately 100 and utilizing bulk database queries reduced downstream API latency by up to 85%.
- Sharding Kafka topics by action type and country prevented slow, high-latency reward workflows in large markets from blocking time-sensitive messaging campaigns in other regions.
- Ensuring the number of Kafka stream partitions is greater than or equal to the maximum consumer server count prevented idle servers during batch processing.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Kafka](https://yomu.fyi/topic/kafka), [Performance](https://yomu.fyi/topic/performance), [Scalability](https://yomu.fyi/topic/scalability)

[Read original post](https://engineering.grab.com/supporting-large-campaigns-at-scale)