Optimally Scaling Kafka Consumer Applications

Summary

Grab's Coban platform runs Golang-based stream processing pipelines on Kubernetes, servicing roughly 400 billion events weekly from Kafka. The initial Horizontal Pod Autoscaler setup caused resource waste and uneven load distribution across Kafka partitions during scale-in and scale-out events. To resolve this, Grab moved to a fixed pod count matching the topic's partition count and adopted Vertical Pod Autoscaling, reducing resource usage versus requests by approximately 45%. The team also introduced Kubernetes priority classes to segment latency-sensitive workloads onto On-Demand nodes and non-critical jobs onto Spot instances. Additionally, overprovisioning via low-priority placeholder pods managed by Cluster Proportional Autoscaler enabled rapid pod rescheduling and reduced deployment delays.

Context

Horizontal Pod Autoscaling on Grab's stream processing platform led to resource wastage from conservative provisioning for peak traffic and resulted in uneven Kafka partition assignment across scaling pods.

Approach / What changed

Grab fixed the number of pipeline pods to match Kafka partition counts retrieved at deployment, replaced HPA with Vertical Pod Autoscaling, separated workloads onto Spot and On-Demand nodes using priority classes and node affinity, and added overprovisioned placeholder pods managed by Cluster Proportional Autoscaler.

Takeaways

• Aligning the number of consumer pods directly with the Kafka topic partition count prevented uneven partition distribution during scaling events.
• Replacing Horizontal Pod Autoscaling with Vertical Pod Autoscaling on fixed-size consumer deployments yielded an approximate 45% reduction in total resource usage relative to requested capacity.
• Using low-priority placeholder pods scaled by Cluster Proportional Autoscaler allowed immediate preemption and rescheduling when Spot instances were terminated.

Related reading

• Grab · Apr 19, 2021

  How We Improved Agent Chat Efficiency with Machine Learning

  Agent typing time represented a large portion of Grab's chat support journey, and 85% of messages were still free typed because agents customized static templates to fit their personal style. To accelerate typing across multilingual markets without robotic templates, Grab built SmartChat, a machine learning feature that provides contextual sentence completion. The team opted for a lightweight seq2seq architecture using single-layered GRU encoder-decoders in TensorFlow instead of bulky attention models to keep model latency under 100ms. The user interface was implemented in React using a content-editable div with inline typeahead suggestions activated via keyboard shortcuts.

  Suman Anand

• Grab · Jul 13, 2021

  App Modularisation at Scale

  Grab transitioned its monolithic mobile application into a modular architecture to resolve increasing code conflicts, slow releases, and difficult team collaboration. The team decomposed the single module by establishing base infrastructure modules, shared UI and utility libraries, discrete feature modules, and bridge kit modules for inter-module communication. Dependency injection using Dagger ties these components together in the main app module while preventing feature modules from directly depending on one another. The architecture spans over 1,000 modules across the app, with more than 200 modules in the Grab Financial Group payments domain where over 95% of modules build in under 15 seconds. This approach accelerated Gradle CI and local builds through parallel compilation and caching, though it increased Gradle sync times, IDE memory usage, and configuration maintenance overhead.

  Amar Jain

• Grab · Jul 29, 2021

  How We Cut GrabFood.com’s Page JavaScript Asset Sizes by 3x

  GrabFood.com faced high cloud costs while serving over 1 TB of weekly network egress across 175 million requests. To minimize egress and improve page performance, the engineering team audited and reduced their webpack dependencies using tools such as webpack-bundle-analyzer and dependency-cruiser. Their strategies included lazy loading rarely used libraries, unifying duplicate modules under single entry points, and substituting utility libraries like axios with native browser Web APIs. Additionally, altering implementation approaches, such as replacing client-side signed JWT cookie encoding with plain JSON strings, eliminated heavy Node dependencies like crypto. These optimizations reduced JavaScript static assets from 750 KB to 250 KB, decreased CloudFront costs by 20%, and accelerated build times by 3.6x.

  Gibson Cheng

• Grab · Dec 3, 2020

  How Grab is Blazing Through the Superapp Bazel Migration

  Grab's mobile superapp scaled past 2.5 million lines of code across both Android and iOS, leading to unsustainable local and CI build times under Gradle and Xcode. To address these bottlenecks, the engineering team analyzed their dependency trees and introduced an internal tool to calculate and optimize the build critical path. They also deployed a Kubernetes-autoscaled remote build system using Mainframer for Android and implemented Test Impact Analysis to run only affected tests in pre-merge validation. While dependency decoupling yielded modest 7% to 10% gains and iOS remote builds proved unscalable on Apple hardware, Android remote builds reduced local compile times by up to 50%, and targeted test execution reduced pre-merge pipeline durations by more than 30%.

  Sergii Grechukha