Scaling Nextdoor’s Datastores: Part 3

Summary

Look-aside caching with object byte serialization can cause critical compatibility failures when application versions, dependencies, or database schemas change. Serialized cache entries bound to specific runtimes risk deserialization errors during deployments, triggering thundering herd queries against the underlying datastore. To resolve this, Nextdoor replaced runtime-bound serialization like Python pickle with MessagePack to serialize Django model objects. The team achieved forward compatibility by letting MessagePack ignore unrecognized new fields in older application code, while backward compatibility relies on mandatory default values for newly added model attributes. Nextdoor prepends a ten-byte header containing format metadata and version information before writing the serialized payload to cache stores.

Context

Caching ORM objects using native byte serialization like Python's pickle tightly couples cached data to specific runtime versions and schema structures. Schema migrations and application deployments can cause deserialization failures, leading to widespread cache misses and thundering herd load on the database.

Approach / What changed

Nextdoor adopted MessagePack to serialize Django Model objects for caching, prepending a 2-byte metadata field and an 8-byte version field to each cache payload. Forward compatibility is achieved by ignoring unknown fields during deserialization, while backward compatibility is handled by requiring default values for new fields.

Takeaways

• MessagePack enables forward compatibility by allowing older application code versions to ignore newly added fields during deserialization.
• Backward compatibility requires developers to define default values so newer code can populate missing attributes when reading older cached entries.
• Nextdoor prepends a 10-byte header consisting of a 2-byte format metadata field and an 8-byte object version field to every cached MessagePack payload.

Related reading

• Nextdoor · Mar 19, 2025

  Scaling Nextdoor’s Datastores: Part 5

  Nextdoor addressed database and cache consistency issues caused by missed cache writes and concurrent read-fill operations in their look-aside architecture. While forward row versioning prevents out-of-order write inconsistencies, writer failures and race conditions during cache misses can leave stale data persisted in Redis. To resolve this, Nextdoor built a reconciliation pipeline that consumes PostgreSQL WAL replication logs with pg-bifrost, streams changes through Apache Kafka, and executes conditional deletions in Redis. The Go-based reconciler operates in two passes using a time wheel, running one pass in near real time and a second pass after a delay exceeding web request timeouts. Because conditional deletion evaluates row versions directly in the cache, the system processes change streams out of order and scales horizontally.

  Slava Markeyev

• Nextdoor · Mar 19, 2025

  Scaling Nextdoor’s Datastores: Part 4