# Exposing a Kafka Cluster via a VPC Endpoint Service

[Grab](https://yomu.fyi/company/grab) · Fabrice Harbulot · Feb 18, 2022

**Type:** Problem & solution

## Summary

To replace VPC peering and reduce attack surfaces, Grab exposed a multi-Availability Zone Apache Kafka cluster in its main AWS VPC to clients in a separate GrabKios VPC using AWS VPC Endpoint Service. Because Kafka requires clients to establish deterministic connections to individual brokers, the team configured a Network Load Balancer with unique TCP ports and dedicated target groups for each broker alongside a shared bootstrap port. They added custom listeners on the Kafka brokers to advertise endpoints using private Route 53 CNAMEs rather than raw interface hostnames. To eliminate unnecessary cross-AZ network latency and data transfer costs, the architecture was refined to advertise AZ-specific private CNAMEs mapped directly to zonal endpoint interfaces.

## Context

Grab needed to deprecate VPC peering to avoid exposing entire VPC networks to each other, moving instead to exposing only specific services across separate AWS accounts while connecting remote GrabKios clients to the main VPC Kafka cluster.

## Approach / What changed

A multi-AZ Network Load Balancer and VPC Endpoint Service were configured with dedicated listeners per broker port and a bootstrap listener, coupled with Route 53 private hosted zones and Kafka advertised listeners configured with zonal CNAMEs.

## Takeaways

- Deterministically addressing individual Kafka brokers through a single Network Load Balancer requires assigning distinct TCP ports and dedicated target groups for each broker.
- Using zonal CNAME records mapped to VPC Endpoint Network Interface zonal hostnames prevents cross-AZ routing during data transfer between clients and brokers.
- Adding new brokers in this architecture introduces operational overhead because it requires allocating new ports, updating NLB listeners, adjusting consumer security groups, and updating Ansible inventories.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [AWS](https://yomu.fyi/topic/aws), [Kafka](https://yomu.fyi/topic/kafka), [Scalability](https://yomu.fyi/topic/scalability), [Streaming](https://yomu.fyi/topic/streaming)

[Read original post](https://engineering.grab.com/exposing-kafka-cluster)