Loading post…
Red HatFederico ValeriAnnouncement
Data liberation: Apache Kafka's native cluster mirroring
Summary
KIP-1279 introduces native cluster mirroring, embedding cross-cluster replication directly into the Kafka broker. This eliminates external tooling, offset translation tables, and recompression, as destination brokers fetch committed records using the standard fetch protocol and append them byte for byte. The approach preserves offsets, compression, and consumer group state, simplifying failover to stopping the mirror and redirecting clients. The architecture uses mirror fetcher threads inside the broker, extending AbstractFetcherThread, and leverages existing client quota mechanisms. Migration involves creating a new KRaft cluster, setting up a mirror, and starting mirroring, with future work including synchronous mirroring and tiered storage integration.
Context
Organizations run multiple Kafka clusters for geographic distribution, compliance boundaries, team isolation, and version segregation. Moving data between clusters previously required external tooling, careful coordination, and operational overhead. KIP-1279 introduces native cluster mirroring directly into the broker, simplifying cross-cluster data replication.
Approach / What changed
KIP-1279 embeds cross-cluster replication directly into the Kafka broker. Mirror fetcher threads run inside the destination broker process, fetching committed records from the source cluster using the standard fetch protocol and appending raw record batches to local partition logs. This eliminates external tooling, offset translation tables, and recompression, preserving offsets, compression, and consumer group state. The architecture uses three main components within each destination broker, extending Kafka’s AbstractFetcherThread. Migration involves standing up a new KRaft cluster, creating a mirror, and starting mirroring.
Takeaways
Cluster mirroring embeds replication directly into the broker, removing the need for external processes and offset translation tables.
Mirror fetcher threads use the standard fetch protocol, appearing as consumer requests, so existing client quota mechanisms apply without modification.
Failover is simplified: stop the mirror and redirect clients, as offsets, compression, and consumer group state are preserved byte for byte.
FAQ
How does cluster mirroring simplify failover?
Cluster mirroring embeds replication into the broker, using mirror fetcher threads that fetch data via the standard fetch protocol and append raw record batches to local logs. This preserves offsets, compression, and consumer group state, making failover as simple as stopping the mirror and redirecting clients.
What is the architecture behind cluster mirroring?
Mirror fetcher threads run inside the destination broker process, fetching data directly from the source cluster using the standard fetch protocol. They append raw record batches to local partition logs, eliminating the need for external processes or offset translation tables.
What future enhancements are planned for cluster mirroring?
Future work includes synchronous mirroring for zero-RPO requirements, tiered storage integration, and support for diskless topics. These enhancements aim to address additional use cases and improve the mirroring capability.
Source and freshness
- Source
- developers.redhat.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.