Scaling Nextdoor’s Datastores: Part 2

Summary

Nextdoor encountered scaling issues after adding read replicas when product engineers were initially tasked with deciding whether to route queries to the primary or replica databases. As business logic grew and gained abstraction layers, engineers struggled to track read-after-write consistency constraints across the call stack. To avoid replication lag race conditions, engineers routinely wrapped logic in database transactions, unintentionally directing all queries to the primary node and eroding read replica benefits over several years. The Core-Services team resolved this by injecting custom tracking logic into their Django ORM layer to monitor table writes during web requests and automate routing. They further optimized the system using a timing-based approach that restored replica read eligibility after the p99.9 replication lag elapsed.

Context

Nextdoor introduced read replicas to manage rising database demand and initially relied on Django ORM routing features to let product engineers manually select primary or replica targets. As codebase abstractions increased, engineers faced read-after-write consistency bugs due to replication lag and began defaulting to primary transactions, which overloaded the primary database.

Approach / What changed

Nextdoor built custom routing logic directly into their ORM to track modified tables within each web request and automatically route subsequent reads. They also implemented a timing-based system that re-enabled read replica routing for modified tables once the p99.9 replication lag duration passed, while cleaning up legacy manual routing and inappropriate transaction usage.

Takeaways

• Manual query routing decisions by product engineers broke down as layered abstractions hid write and read operations across call stacks.
• Wrapping operations in transactions to avoid replication lag routed all enclosed queries to the primary, eroding replica capacity gains over years.
• Injecting table-modification tracking into the ORM and restoring replica eligibility after p99.9 replication lag successfully offloaded read traffic from the primary.

Related reading

• Nextdoor · Mar 19, 2025

  Scaling Nextdoor’s Datastores: Part 5

  Nextdoor addressed database and cache consistency issues caused by missed cache writes and concurrent read-fill operations in their look-aside architecture. While forward row versioning prevents out-of-order write inconsistencies, writer failures and race conditions during cache misses can leave stale data persisted in Redis. To resolve this, Nextdoor built a reconciliation pipeline that consumes PostgreSQL WAL replication logs with pg-bifrost, streams changes through Apache Kafka, and executes conditional deletions in Redis. The Go-based reconciler operates in two passes using a time wheel, running one pass in near real time and a second pass after a delay exceeding web request timeouts. Because conditional deletion evaluates row versions directly in the cache, the system processes change streams out of order and scales horizontally.

  Slava Markeyev

• Nextdoor · Mar 19, 2025

  Scaling Nextdoor’s Datastores: Part 1