---
title: "Building Zone Failure Resilience in Apache Pinot™ at Uber"
description: "Uber describes how it made Apache Pinot clusters resilient to zone failures while preserving query and ingestion availability for Tier-0 use cases. It maps Uber’s virtual resource isolation group IDs to Pinot server pools, uses pool-based instance assignment with replica-group segment assignment, and requires at least as many isolation groups as the table’s replication factor. During migration, automated pool registration, configuration backfills, granular APIs, selective rebalancing, and rollback support were used across more than 400 Tier 0-1 clusters and thousands of tables. An internal drill found that one remaining replica continued serving traffic without query degradation after a zone went down, while isolation-group-based claim and release policies enabled parallel restarts and sequential rollouts with baking time."
---

# Building Zone Failure Resilience in Apache Pinot™ at Uber

[Uber](https://yomu.fyi/company/uber) · Jul 2, 2026

**Type:** Problem & solution

## Summary

Uber describes how it made Apache Pinot clusters resilient to zone failures while preserving query and ingestion availability for Tier-0 use cases. It maps Uber’s virtual resource isolation group IDs to Pinot server pools, uses pool-based instance assignment with replica-group segment assignment, and requires at least as many isolation groups as the table’s replication factor. During migration, automated pool registration, configuration backfills, granular APIs, selective rebalancing, and rollback support were used across more than 400 Tier 0-1 clusters and thousands of tables. An internal drill found that one remaining replica continued serving traffic without query degradation after a zone went down, while isolation-group-based claim and release policies enabled parallel restarts and sequential rollouts with baking time.

## Context

Uber needed Apache Pinot to withstand zone failures without affecting queries or ingestion across Tier-0 real-time analytics workloads. Existing tag-based instance assignment and balanced segment assignment distributed data within tenants but did not guarantee that instances or replicas occupied different physical zones. The rollout also had to cover more than 400 Tier 0-1 clusters and thousands of tables without degrading live traffic.

## Approach / What changed

The implementation maps Uber Odin isolation group IDs to Pinot server pool numbers, then combines pool-based instance assignment with replica-group segment assignment to distribute replicas across isolation groups. Odin workers automatically register pools with Pinot controllers. Migration uses configuration backfills, granular APIs with auditing and rollback support, and selective table rebalancing. Isolation-group-based claim and release policies also coordinate parallel operations and staged rollouts with baking time.

## Takeaways

- Pool-based instance assignment combined with replica-group segment assignment places segment replicas in different server pools, which correspond to zones; remaining replicas can continue serving when a pool fails.
- Pinot requires the configured isolation-group count to be at least the table’s replication factor. Uber generally uses two isolation groups to match its default replication factor of two.
- Migration selectively rebalances upsert tables and tables with retention longer than six months, while other tables receive new assignments as older segments are purged; internal APIs support table-, tenant-, and region-level operations.

**Tags:** [Apache Pinot](https://yomu.fyi/topic/apache-pinot), [Release Engineering](https://yomu.fyi/topic/release-engineering), [Reliability](https://yomu.fyi/topic/reliability)

- Source: [Uber](https://www.uber.com/us/en/blog/building-zone-failure-resilience-in-apache-pinot-at-uber)
- Source URL: https://www.uber.com/us/en/blog/building-zone-failure-resilience-in-apache-pinot-at-uber
- Ingested by Yomu: 2026-09-11T04:02:01.597Z

[Read original post](https://www.uber.com/us/en/blog/building-zone-failure-resilience-in-apache-pinot-at-uber)