# Zero traffic cost for Kafka consumers

[Grab](https://yomu.fyi/company/grab) · Fabrice Harbulot · Jul 7, 2023

**Type:** Problem & solution

## Summary

Grab's real-time data streaming platform team reduced infrastructure expenses by enabling Kafka consumers to fetch from the closest partition replica instead of partition leaders. The platform previously operated across three AWS Availability Zones with three-way replication, resulting in cross-AZ network fees that constituted half of the Kafka platform's total cost. To address this, the team performed a zero-downtime upgrade to Kafka 3.1, configured a rack-aware replica selector using AWS Availability Zone IDs, and updated consumer SDKs to populate the client rack parameter. The rollout produced a 25% drop in cross-AZ traffic within three months under steady data volumes. However, the change increased end-to-end latency by up to 500ms, created potential CPU load skews across zones, and removed graceful isolation during broker rotation maintenance.

## Context

Kafka consumers by default fetch data exclusively from partition leaders, causing roughly two-thirds of consumer traffic to cross Availability Zone boundaries in a multi-AZ AWS setup and generating half of Grab's total Kafka platform costs.

## Approach / What changed

Upgraded Kafka clusters to version 3.1 via a zero-downtime phased protocol rollout, configured brokers with the RackAwareReplicaSelector using EC2 AZ IDs, and updated consumer SDKs and connectors to dynamically set client.rack.

## Takeaways

- Enabling consumers to fetch from local AZ replicas reduced Grab's cross-AZ network traffic by 25% over a three-month period.
- Fetching from follower replicas increased end-to-end producer-to-consumer latency by up to 500ms, requiring latency-sensitive workloads to retain the traditional leader-fetching design.
- Demoting a broker for maintenance fails to gracefully isolate it from consumers reading follower replicas, requiring consumers to implement connection error handling and retry mechanisms.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [AWS](https://yomu.fyi/topic/aws), [Kafka](https://yomu.fyi/topic/kafka), [Streaming](https://yomu.fyi/topic/streaming)

[Read original post](https://engineering.grab.com/zero-traffic-cost)