Scaling Nextdoor’s Datastores: Part 4

Summary

Look-aside caching systems can become inconsistent when concurrent database updates execute cache writes out of order, allowing stale data to overwrite newer modifications. To prevent these racing writes, Nextdoor introduced a unique, monotonic db_version column to Postgres tables using database triggers that initialize version numbers on insert and increment them on update. Application updates retrieve this new version inside a transaction block and attach it as a metadata header to serialized cache values. Redis then executes custom Lua scripts, specifically set_if_version and del_if_version, to perform atomic conditional updates that reject any incoming payload with a version lower than or equal to the stored version. This serializable check ensures that out-of-order writes are dropped and the cache remains strictly aligned with the latest database state.

Context

In distributed look-aside caching systems, concurrent web requests can execute database updates and subsequent cache writes in different orders. When an earlier database write updates the cache after a later write has already finished, stale data overwrites fresh data, causing the cache and database to become inconsistent.

Approach / What changed

The team added a unique, monotonic db_version column to database tables, populated via a Postgres BEFORE INSERT OR UPDATE trigger. Because Django's ORM does not return updated rows, the application executes a select within the same transaction to obtain the new version. The version is attached as a metadata header on the serialized cache object. On the cache layer, Redis executes custom Lua scripts (set_if_version and del_if_version) to perform atomic compare-and-set operations, rejecting updates when the incoming version is equal to or lower than the stored version.

Takeaways

• Timestamp-based versioning is insufficient for cache consistency because timestamps are neither guaranteed to be strictly monotonic nor globally unique across concurrent writers.
• Row version numbers must be generated atomically on the database side via mechanisms like database triggers or user-defined functions rather than client-side logic to ensure monotonic consistency.
• Running custom Lua scripts inside single-threaded Redis enables atomic conditional updates and deletes, preventing race conditions that occur when version checks happen on the client side.

Related reading

• Nextdoor · Mar 19, 2025

  Scaling Nextdoor’s Datastores: Part 5

  Nextdoor addressed database and cache consistency issues caused by missed cache writes and concurrent read-fill operations in their look-aside architecture. While forward row versioning prevents out-of-order write inconsistencies, writer failures and race conditions during cache misses can leave stale data persisted in Redis. To resolve this, Nextdoor built a reconciliation pipeline that consumes PostgreSQL WAL replication logs with pg-bifrost, streams changes through Apache Kafka, and executes conditional deletions in Redis. The Go-based reconciler operates in two passes using a time wheel, running one pass in near real time and a second pass after a delay exceeding web request timeouts. Because conditional deletion evaluates row versions directly in the cache, the system processes change streams out of order and scales horizontally.

  Slava Markeyev

• Nextdoor · Mar 19, 2025

  Scaling Nextdoor’s Datastores: Part 3