Scaling Nextdoor’s Datastores: Part 5

Summary

Nextdoor addressed database and cache consistency issues caused by missed cache writes and concurrent read-fill operations in their look-aside architecture. While forward row versioning prevents out-of-order write inconsistencies, writer failures and race conditions during cache misses can leave stale data persisted in Redis. To resolve this, Nextdoor built a reconciliation pipeline that consumes PostgreSQL WAL replication logs with pg-bifrost, streams changes through Apache Kafka, and executes conditional deletions in Redis. The Go-based reconciler operates in two passes using a time wheel, running one pass in near real time and a second pass after a delay exceeding web request timeouts. Because conditional deletion evaluates row versions directly in the cache, the system processes change streams out of order and scales horizontally.

Context

Forward cache versioning handles racing writes but fails if writers cannot communicate with the cache or if a reader fills stale data during an in-flight write. When a write to the database succeeds but fails to update the cache, the cache becomes inconsistent unless an external process intervenes. Additionally, concurrent reads during missed writes can overwrite fresh invalidations with stale database snapshots.

Approach / What changed

Nextdoor built an asynchronous reconciliation pipeline consuming PostgreSQL WAL logs via pg-bifrost and Apache Kafka. A Go-based reconciler processes the change stream and executes conditional cache deletions (del_if_version) in Redis based on row versions. To address cache-fill race conditions, the system uses a two-pass reconciliation approach implemented with a time wheel: one pass applies invalidations in near real time, while a second pass runs with a fixed delay slightly higher than the web request timeout.

Takeaways

• Using database row versions for conditional cache deletions (del_if_version) removes the requirement for strictly ordered change stream processing, enabling horizontal scaling of reconciler workers.
• A two-pass reconciliation mechanism mitigates race conditions during cache miss repopulation by executing one invalidation pass immediately and a second pass delayed beyond the web request timeout.
• The reconciliation pipeline combines pg-bifrost to capture PostgreSQL WAL logs, Apache Kafka as a persistent message bus, and a Go-based consumer issuing conditional Redis deletions.

Related reading

• Nextdoor · Mar 19, 2025

  Scaling Nextdoor’s Datastores: Part 4

  Look-aside caching systems can become inconsistent when concurrent database updates execute cache writes out of order, allowing stale data to overwrite newer modifications. To prevent these racing writes, Nextdoor introduced a unique, monotonic db_version column to Postgres tables using database triggers that initialize version numbers on insert and increment them on update. Application updates retrieve this new version inside a transaction block and attach it as a metadata header to serialized cache values. Redis then executes custom Lua scripts, specifically set_if_version and del_if_version, to perform atomic conditional updates that reject any incoming payload with a version lower than or equal to the stored version. This serializable check ensures that out-of-order writes are dropped and the cache remains strictly aligned with the latest database state.

  Ronak Shah

• Nextdoor · Mar 19, 2025

  Scaling Nextdoor’s Datastores: Part 3