Apache Kafka 0.10 Evaluating Performance in Distributed Systems

Summary

Testing Apache Kafka 0.10 revealed an apparent 33% throughput decline versus Kafka 0.9, prompting an investigation into whether the regression affected different plans, connection types, message sizes, and workload scales. Heroku’s benchmark system used multiple producer and consumer dynos, while the team applied Brendan Gregg’s USE method to examine resource utilization, saturation, and errors. The diagnosis was network-card saturation: Kafka 0.10 adds an 8-byte timestamp to every message, which at 1.5 million messages per second and replication factor 3 produced 288 Mbps of additional cluster traffic, causing packet drops and retransmits. Reproducing the effect by adding 8 bytes to Kafka 0.9 confirmed the explanation, and the post says most production workloads are unlikely to be affected, though clusters near network limits may be.

Takeaways

1. 1

   Kafka 0.10’s 8-byte per-message timestamp added 288 Mbps of cluster traffic at 1.5 million messages per second with replication factor 3.

2. 2

   Network-card oversaturation caused packet drops and retransmits, sharply reducing throughput in the benchmark.

3. 3

   The post recommends avoiding network saturation, contributed documentation about the added traffic, and says Kafka 0.9.0.1 remained the recommended production version for Heroku Kafka at the time.