---
title: "Managing Elasticsearch Reindex at Scale: Performance, Reliability, and Observability"
description: "Palantir’s document store needs to repair or rebuild Elasticsearch indices when mappings, settings, cluster versions, or index state change, while continuing to serve live search and writes. Its reindex machinery reconstructs schema-aware indices from the strongly consistent database, rather than relying on Elasticsearch’s _reindex API, and is designed to limit manual work and resource consumption across many deployments and schemas. For an online reindex, it creates a physical shadow index with the intended mappings and settings, backfills it from the database, dual-writes current API changes to both indices, and promotes the shadow index after completion; searches remain on the primary during the rebuild. Cross-cluster migrations use a separate machine with cluster-specific metadata, configuration, index and alias evolution, and preservation of old-cluster indices to support rollback."
---

# Managing Elasticsearch Reindex at Scale: Performance, Reliability, and Observability

[Palantir](https://yomu.fyi/company/palantir) · Palantir · Jun 8, 2026

**Type:** Explainer

## Summary

Palantir’s document store needs to repair or rebuild Elasticsearch indices when mappings, settings, cluster versions, or index state change, while continuing to serve live search and writes. Its reindex machinery reconstructs schema-aware indices from the strongly consistent database, rather than relying on Elasticsearch’s \_reindex API, and is designed to limit manual work and resource consumption across many deployments and schemas. For an online reindex, it creates a physical shadow index with the intended mappings and settings, backfills it from the database, dual-writes current API changes to both indices, and promotes the shadow index after completion; searches remain on the primary during the rebuild. Cross-cluster migrations use a separate machine with cluster-specific metadata, configuration, index and alias evolution, and preservation of old-cluster indices to support rollback.

## Context

Elasticsearch indices may require rebuilding when schemas or mappings evolve, index settings change, indices become inconsistent or corrupted, or services migrate between clusters. Reindexing must support many schemas and deployments without taking services offline, disrupting live search, or requiring extensive operator intervention.

## Approach / What changed

The document store orchestrates schema-aware reindexing from the strongly consistent database. It creates a shadow index with the desired mappings and settings, backfills it, dual-writes live API changes to the primary and shadow indices, and promotes the shadow when ready. Cross-cluster migration uses a separate reindex machine with cluster-specific configuration and preserves old-cluster indices for rollback.

## Takeaways

- A shadow index receives backfilled data and live dual-writes while searches continue using the existing primary index; it is promoted after reindexing completes.
- Reindexing is sourced from the document store’s strongly consistent database rather than Elasticsearch’s \_reindex API, so the document store orchestrates the process.
- Primary-cluster repair and cross-cluster migration use separate reindex machines, simplifying cluster-specific configuration, metadata management, code paths, and observability while retaining rollback indices.

**Tags:** [Elasticsearch](https://yomu.fyi/topic/elasticsearch), [Migrations](https://yomu.fyi/topic/migration), [Performance](https://yomu.fyi/topic/performance), [Reliability](https://yomu.fyi/topic/reliability), [Scalability](https://yomu.fyi/topic/scalability)

- Source: [Palantir](https://blog.palantir.com/managing-elasticsearch-reindex-at-scale-performance-reliability-and-observability-cf948d0efd47)
- Source URL: https://blog.palantir.com/managing-elasticsearch-reindex-at-scale-performance-reliability-and-observability-cf948d0efd47
- Ingested by Yomu: 2026-08-30T13:12:55.945Z

[Read original post](https://blog.palantir.com/managing-elasticsearch-reindex-at-scale-performance-reliability-and-observability-cf948d0efd47)