---
title: "Rebuilding Notion’s lexical search reindexer"
description: "Notion’s lexical search reindexer had become a multi-week, manually monitored process that produced about 90% data consistency. The old design also relied on Snowflake metadata lookups and a separate catchup pipeline. The redesign replaces ECS/Fargate workers and live Elasticsearch writes with an Apache Spark–native pipeline that routes documents to exact index-and-shard pairs and generates native snapshot files through embedded Elasticsearch nodes. Temporary indices capture live updates while the snapshot is built, after which aliases, external document versions, Elasticsearch’s _reindex, and phantom detection handle catchup, including alive-to-deleted transitions. The new system completes full reindexing in under two days, catchup in under an hour, reaches 100% consistency, removes Snowflake and ECS from the indexing path, and reduces manual intervention to under two hours per run."
---

# Rebuilding Notion’s lexical search reindexer

[Notion](https://yomu.fyi/company/notion) · Aravind Selvan, Calder Lund, and Gaurang Sadekar · Aug 10, 2026

**Type:** Problem & solution

## Summary

Notion’s lexical search reindexer had become a multi-week, manually monitored process that produced about 90% data consistency. The old design also relied on Snowflake metadata lookups and a separate catchup pipeline. The redesign replaces ECS/Fargate workers and live Elasticsearch writes with an Apache Spark–native pipeline that routes documents to exact index-and-shard pairs and generates native snapshot files through embedded Elasticsearch nodes. Temporary indices capture live updates while the snapshot is built, after which aliases, external document versions, Elasticsearch’s \_reindex, and phantom detection handle catchup, including alive-to-deleted transitions. The new system completes full reindexing in under two days, catchup in under an hour, reaches 100% consistency, removes Snowflake and ECS from the indexing path, and reduces manual intervention to under two hours per run.

## Context

Notion periodically rebuilds its lexical search index to backfill existing blocks for new searchable features, change text analysis, or support Elasticsearch migrations and new regions. The previous process took more than two weeks, required ongoing monitoring, depended on Snowflake and ECS, and achieved about 90% consistency.

## Approach / What changed

The redesign removes Snowflake lookups from the indexing hot path, partitions documents according to live index and shard routing, generates Elasticsearch-native snapshots with embedded Elasticsearch nodes in Spark tasks, and stores them in S3. Temporary indices capture live writes during the build; aliases, external document versions, \_reindex, and phantom detection complete catchup safely.

## Takeaways

- A Spark partitioner applies both application-level index assignment and Elasticsearch’s shard-ID hash, producing documents partitioned identically to direct live writes.
- Snapshot generation bypasses Elasticsearch’s live indexing API: embedded Elasticsearch nodes create per-shard manifests and Lucene segments, which a Hadoop OutputCommitter uploads to S3.
- External document versions make catchup retryable, while a separate phantom-detection step handles blocks that move between alive and deleted index families.

**Tags:** [Apache Spark](https://yomu.fyi/topic/apache-spark), [Elasticsearch](https://yomu.fyi/topic/elasticsearch), [S3](https://yomu.fyi/topic/s3)

- Source: [Notion](https://www.notion.com/en-gb/blog/rebuilding-notions-lexical-search-reindexer)
- Source URL: https://www.notion.com/en-gb/blog/rebuilding-notions-lexical-search-reindexer
- Ingested by Yomu: 2026-09-09T05:19:29.779Z

[Read original post](https://www.notion.com/en-gb/blog/rebuilding-notions-lexical-search-reindexer)