---
title: "Zone-Failure-Resilient OpenSearch® at Uber"
description: "Uber describes how it made OpenSearch deployments zone-failure resilient so querying and ingestion remain available after a complete zone loss. The design combines OpenSearch’s shard-allocation awareness with Odin-based isolation groups that distribute nodes predictably across failure domains and balance roles. Clusters maintain at least three shard copies across three isolation groups, while forced awareness keeps shards for a missing group unassigned instead of triggering heavy rebalancing; this trades immediate full replication for stability in surviving zones. For manager resiliency, five cluster manager nodes and cluster.auto_shrink_voting_configuration allow quorum after a zone failure plus one additional manager failure. Uber reports that isolation groups replaced uneven physical zone IDs, enabling complete shard assignment, preventing disk skew and hot nodes, and supporting green cluster health."
---

# Zone-Failure-Resilient OpenSearch® at Uber

[Uber](https://yomu.fyi/company/uber) · Jul 21, 2026

**Type:** Problem & solution

## Summary

Uber describes how it made OpenSearch deployments zone-failure resilient so querying and ingestion remain available after a complete zone loss. The design combines OpenSearch’s shard-allocation awareness with Odin-based isolation groups that distribute nodes predictably across failure domains and balance roles. Clusters maintain at least three shard copies across three isolation groups, while forced awareness keeps shards for a missing group unassigned instead of triggering heavy rebalancing; this trades immediate full replication for stability in surviving zones. For manager resiliency, five cluster manager nodes and cluster.auto\_shrink\_voting\_configuration allow quorum after a zone failure plus one additional manager failure. Uber reports that isolation groups replaced uneven physical zone IDs, enabling complete shard assignment, preventing disk skew and hot nodes, and supporting green cluster health.

## Context

Uber requires OpenSearch clusters to withstand a complete zone failure without losing querying or data-ingestion capability. Physical zones can have uneven node counts, causing invalid shard placements, yellow cluster states, disk skew, and hot nodes. The design also addresses a zone failure followed by an additional node failure in another zone.

## Approach / What changed

Uber combines OpenSearch shard allocation awareness with Odin-based isolation groups that map nodes to balanced failure domains and maintain role-level distribution. Forced shard allocation awareness prevents aggressive redistribution when a group disappears. At least three shard copies and five cluster manager nodes, together with cluster.auto\_shrink\_voting\_configuration, preserve data availability and manager quorum during zone-plus-one failures.

## Takeaways

- Isolation groups abstract uneven physical zones into three balanced groups, ensuring even node distribution and preventing shard-placement problems caused by physical zone asymmetry.
- Forced shard allocation awareness leaves shards for a failed isolation group unassigned rather than overloading surviving groups; recovery occurs when the group returns or an administrator changes the configuration.
- With five cluster manager nodes, voting configuration can shrink from five to three after a zone failure, then retain quorum with two nodes after one additional manager failure.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Distributed Systems](https://yomu.fyi/topic/distributed-systems), [High Availability](https://yomu.fyi/topic/high-availability), [OpenSearch](https://yomu.fyi/topic/opensearch)

- Source: [Uber](https://www.uber.com/us/en/blog/zone-failure-resilient)
- Source URL: https://www.uber.com/us/en/blog/zone-failure-resilient
- Ingested by Yomu: 2026-09-10T16:01:24.601Z

[Read original post](https://www.uber.com/us/en/blog/zone-failure-resilient)