Scaling Nextdoor’s Datastores: Part 4

Summary

Look-aside caching systems can become inconsistent when concurrent database updates execute cache writes out of order, allowing stale data to overwrite newer modifications. To prevent these racing writes, Nextdoor introduced a unique, monotonic db_version column to Postgres tables using database triggers that initialize version numbers on insert and increment them on update. Application updates retrieve this new version inside a transaction block and attach it as a metadata header to serialized cache values. Redis then executes custom Lua scripts, specifically set_if_version and del_if_version, to perform atomic conditional updates that reject any incoming payload with a version lower than or equal to the stored version. This serializable check ensures that out-of-order writes are dropped and the cache remains strictly aligned with the latest database state.

Takeaways

1. 1

   Timestamp-based versioning is insufficient for cache consistency because timestamps are neither guaranteed to be strictly monotonic nor globally unique across concurrent writers.

2. 2

   Row version numbers must be generated atomically on the database side via mechanisms like database triggers or user-defined functions rather than client-side logic to ensure monotonic consistency.

3. 3

   Running custom Lua scripts inside single-threaded Redis enables atomic conditional updates and deletes, preventing race conditions that occur when version checks happen on the client side.