Pulling the Thread on Kafka’s Compacted Topics

Summary

Apache Kafka’s log cleaner for compacted topics could crash silently when its offset map lacked enough space to hold a full log segment. Because the broker continued running without that internal thread, superseded data accumulated, threatening node and potentially cluster stability. The post explains compaction’s two phases: building an offset map for newer messages, then rewriting segments while retaining the latest offset for each key, with the latest segment excluded from the map. Kafka was changed to remove the crashing assertion and track partial segment loads so cleaning can recover without fitting an entire segment in memory. The roughly 100-line patch shipped in Kafka 0.10.1.1, which the post identifies as Heroku’s default version, and prevents the cleaner from dying silently.

Takeaways

1. 1

   Compacted topics retain the newest message for each key, using periodic cleaning and optional time-based retention; duplicate keys may remain until cleaning completes.

2. 2

   Kafka’s cleaner builds an offset map for new messages, then rewrites older segments while removing messages whose offsets are lower than the mapped offset.

3. 3

   The fix was committed to Kafka and released in version 0.10.1.1, allowing the cleaner to handle partial segments without dying silently.