Zero traffic cost for Kafka consumers

Summary

Grab's real-time data streaming platform team reduced infrastructure expenses by enabling Kafka consumers to fetch from the closest partition replica instead of partition leaders. The platform previously operated across three AWS Availability Zones with three-way replication, resulting in cross-AZ network fees that constituted half of the Kafka platform's total cost. To address this, the team performed a zero-downtime upgrade to Kafka 3.1, configured a rack-aware replica selector using AWS Availability Zone IDs, and updated consumer SDKs to populate the client rack parameter. The rollout produced a 25% drop in cross-AZ traffic within three months under steady data volumes. However, the change increased end-to-end latency by up to 500ms, created potential CPU load skews across zones, and removed graceful isolation during broker rotation maintenance.

Takeaways

1. 1

   Enabling consumers to fetch from local AZ replicas reduced Grab's cross-AZ network traffic by 25% over a three-month period.

2. 2

   Fetching from follower replicas increased end-to-end producer-to-consumer latency by up to 500ms, requiring latency-sensitive workloads to retain the traditional leader-fetching design.

3. 3

   Demoting a broker for maintenance fails to gracefully isolate it from consumers reading follower replicas, requiring consumers to implement connection error handling and retry mechanisms.