Sharding etcd on hosted control planes and what the numbers actually say

Summary

Sharding etcd on hosted control planes splits the key-value store into multiple instances by resource kind, a tech preview feature tested with a main shard for cluster state and a dedicated events shard on Red Hat OpenShift 4.21 with HyperShift. Under event pressure from synthetic storms and realistic crashlooping pods, the sharded configuration showed up to 96% fewer slow applies, faster compaction and raft snapshot durations, and 60% more WAL fsyncs and 90% more backend commits combined. The sharded arm never failed the event storm workload, while the standard arm occasionally did. Pod-ready latencies and workload finish times remained comparable across both configurations. The trade-offs are roughly 1.8 to 2.3 times the etcd container memory and a compaction regression under large custom resource quota pressure that is still under investigation.

Takeaways

1. 1

   Sharding reduces slow applies by up to 96% under event pressure, with faster compaction and raft snapshots.

2. 2

   Sharding increases total write throughput (60% more WAL fsyncs, 90% more backend commits) but costs 1.8–2.3x etcd container memory.

3. 3

   The only regression is a compaction slowdown under large custom resource quota pressure, still under investigation.

FAQ