Exposing a Kafka Cluster via a VPC Endpoint Service

Summary

To replace VPC peering and reduce attack surfaces, Grab exposed a multi-Availability Zone Apache Kafka cluster in its main AWS VPC to clients in a separate GrabKios VPC using AWS VPC Endpoint Service. Because Kafka requires clients to establish deterministic connections to individual brokers, the team configured a Network Load Balancer with unique TCP ports and dedicated target groups for each broker alongside a shared bootstrap port. They added custom listeners on the Kafka brokers to advertise endpoints using private Route 53 CNAMEs rather than raw interface hostnames. To eliminate unnecessary cross-AZ network latency and data transfer costs, the architecture was refined to advertise AZ-specific private CNAMEs mapped directly to zonal endpoint interfaces.

Context

Grab needed to deprecate VPC peering to avoid exposing entire VPC networks to each other, moving instead to exposing only specific services across separate AWS accounts while connecting remote GrabKios clients to the main VPC Kafka cluster.

Approach / What changed

A multi-AZ Network Load Balancer and VPC Endpoint Service were configured with dedicated listeners per broker port and a bootstrap listener, coupled with Route 53 private hosted zones and Kafka advertised listeners configured with zonal CNAMEs.

Takeaways

• Deterministically addressing individual Kafka brokers through a single Network Load Balancer requires assigning distinct TCP ports and dedicated target groups for each broker.
• Using zonal CNAME records mapped to VPC Endpoint Network Interface zonal hostnames prevents cross-AZ routing during data transfer between clients and brokers.
• Adding new brokers in this architecture introduces operational overhead because it requires allocating new ports, updating NLB listeners, adjusting consumer security groups, and updating Ansible inventories.

Related reading

• Grab · Nov 23, 2020

  Democratising Fare Storage at Scale Using Event Sourcing

  Grab's legacy system stored booking and fare details in a single relational table, creating a bloated booking entity that tracked only the latest fare state and hindered rapid feature iteration. To resolve scalability, stability, and debugging challenges across millions of daily bookings, the team developed Fare Storage using the Event Sourcing pattern. The new architecture persists all fare modification events chronologically in DynamoDB, backed by a cache for eventually consistent reads and message streaming for downstream processing. The platform employs optimistic locking with versioning to manage concurrent updates, enforces idempotency through client-generated transaction UUIDs, and delegates metadata serialization to an SDK to prevent storage API changes.

  Sourabh Suman

• Grab · Jan 26, 2021

  Serving Driver-partners Data at Scale Using Mirror Cache

  Grab's Drivers Data service handles up to 10,000 requests per second during peak hours to supply driver information across backend microservices. The original setup used MySQL with Redis and standalone in-memory local caches, but yielded a low 25% local cache hit rate due to traffic patterns characterized by high burst frequency for individual drivers alongside redundant database calls across nodes. To solve this, the team developed Mirror Cache, an in-memory caching system that pairs Dgraph's Ristretto library with an asynchronous gRPC replication layer to mirror updates across cluster nodes. The replicator batches updates within the same AWS availability zone and forwards data to single nodes across zones to minimize transfer overhead. Production deployment increased the in-memory cache hit rate to approximately 75% and reduced direct MySQL queries by 5%.

  Indrajit Sarkar

• Grab · Jun 30, 2021

  Debugging High Latency Due to Context Leaks

  Market-Store, Grab's feature store for real-time machine learning features, experienced latency spikes from under 200 milliseconds to 2 seconds as traffic grew. Metrics and logs showed no direct correlation to API issues, but heap profiling with PPROF revealed continuously increasing memory held by child contexts. Further analysis tracked the leak to an update in Grab's open-source Async Library, which switched background contexts to uncancelled task contexts for worker runners. Because parent contexts maintained references to these uncancelled child contexts, the garbage collector could not reclaim their memory. This progressive memory exhaustion directly degraded API latency.

  Sourabh Suman

• Grab · Sep 1, 2021

  Automating Multi-Armed Bandit testing during feature rollout

  Traditional feature rollouts and Multi-Armed Bandit testing operate as separate workflows that often depend on delayed offline analysis. To eliminate manual intervention, the Multi-Armed Bandit Optimiser automates testing concurrently during feature rollouts by responding to minute-level feedback metrics. The architecture connects Kafka Streams data processing, a metrics server with Spark jobs, and an adaptive rollout module updating online experimentation configurations. Candidate models are evaluated via Thompson Sampling on Beta distributions, with Monte Carlo simulations determining traffic allocation across user entities. In production for the GrabFood recommendation widget, the system optimizes the Effective Conversion Rate over a 30-minute window and includes fallback distribution logic.

  Weicheng Zhu