Zone-Failure-Resilient OpenSearch® at Uber

Summary

Uber describes how it made OpenSearch deployments zone-failure resilient so querying and ingestion remain available after a complete zone loss. The design combines OpenSearch’s shard-allocation awareness with Odin-based isolation groups that distribute nodes predictably across failure domains and balance roles. Clusters maintain at least three shard copies across three isolation groups, while forced awareness keeps shards for a missing group unassigned instead of triggering heavy rebalancing; this trades immediate full replication for stability in surviving zones. For manager resiliency, five cluster manager nodes and cluster.auto_shrink_voting_configuration allow quorum after a zone failure plus one additional manager failure. Uber reports that isolation groups replaced uneven physical zone IDs, enabling complete shard assignment, preventing disk skew and hot nodes, and supporting green cluster health.

Takeaways

1. 1

   Isolation groups abstract uneven physical zones into three balanced groups, ensuring even node distribution and preventing shard-placement problems caused by physical zone asymmetry.

2. 2

   Forced shard allocation awareness leaves shards for a failed isolation group unassigned rather than overloading surviving groups; recovery occurs when the group returns or an administrator changes the configuration.

3. 3

   With five cluster manager nodes, voting configuration can shrink from five to three after a zone failure, then retain quorum with two nodes after one additional manager failure.