From Pre-trained to Fine-tuned: Nextdoor’s Path to Effective Embedding Applications

Summary

Nextdoor transitioned its ranking and recommendation pipelines from traditional continuous and discrete interaction features to transformer-based representation learning. The engineering team deployed pre-trained Sentence-BERT models to generate multilingual post and comment representations, which were aggregated daily by interaction type to form user embeddings. To improve search recall, the team fine-tuned sentence transformers on unlabeled query session logs using contrastive learning and integrated HNSWlib for approximate nearest neighbor retrieval. Subsequent iterations incorporated labeled feedback, BERTopic for coarse personalization, and experiments with CLIP image embeddings. Infrastructure scaling challenges were addressed by performing embedding transformations directly within FeatureStore and optimizing feature payload formats to minimize microservice network bandwidth.

Context

Nextdoor relied primarily on continuous and discrete features from interaction logs for ML ranking systems like feed ranking and notifications scoring. These models lacked deep content understanding from text and images, requiring transformer-based representation learning to capture nuanced user signals and improve personalization and local search.

Approach / What changed

Nextdoor implemented entity embedding models across three phases: using pre-trained Sentence-BERT for multilingual text representations and weighted user aggregations, fine-tuning search query embeddings using self-supervised contrastive learning and HNSWlib, and applying labeled fine-tuning along with BERTopic and CLIP. Transformations were moved directly into FeatureStore to lower serving costs.

Takeaways

• Pre-trained multilingual Sentence-BERT models generated post and comment embeddings that were aggregated daily by interaction weight into user embeddings, boosting notification and feed ranking metrics.
• Self-supervised contrastive learning on search query logs paired with HSNWlib approximate nearest neighbor search improved search recall, reduced null queries, and sped up query expansion latencies by more than 10x.
• Transforming high-dimensional embedding vectors directly within FeatureStore reduced microservice network bandwidth, while payload format optimization and caching helped control dimensionality and serving costs.

Related reading

• Nextdoor · Jul 6, 2022

  Typeahead Search at Nextdoor

  Nextdoor built a proximity-based autocomplete service to power typeahead search and mention features across its hyperlocal platform for hundreds of millions of entities, including users and businesses. The system shards geographic data using Uber's open-source H3 geohashing library and stores prefix indexes in memory using Redis sorted sets. By adopting a Command Query Responsibility Segregation architecture, ingestion writes are processed on Redis primary nodes and replicated to read-only search nodes with under 10 milliseconds of replication lag. Dedicated APIs handle indexing, typeahead lookups, and ranking before returning hydrated results. Operating since August 2021, the service processes hundreds of millions of monthly typeahead queries while maintaining a P95 search latency below 30 milliseconds.

  Jerry Tian

• huggingface.co · Nov 9, 2020

  Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models

  Pre-training sequence-to-sequence transformer models incurs massive computational costs, limiting development primarily to large institutions. To mitigate these expenses, encoder-decoder architectures can be warm-started using existing pre-trained checkpoints from encoder-only or decoder-only models like BERT and GPT-2. This walkthrough details the methodology and implementation of warm-starting sequence-to-sequence architectures using Hugging Face Transformers. By utilizing the EncoderDecoderModel framework alongside Seq2SeqTrainer, practitioners can construct and fine-tune models such as BERT2BERT on datasets like CNN/DailyMail. The resulting fully trained BERT2BERT model achieves a ROUGE-2 score of 18.22 on the full evaluation set, matching competitive sequence generation baselines at a fraction of standard pre-training costs.