---
title: "Let AI Entertain You: Increasing User Engagement with Generative AI and Rejection Sampling"
description: "Nextdoor implemented a generative AI framework to improve email notification subject lines for its New and Trending emails. Using off-the-shelf ChatGPT API outputs directly resulted in marketing-like phrases, hallucinations, and lower click-through rates compared to original user-generated post snippets. To resolve this, the team prompted the language model strictly to extract authentic post phrases without rewriting, eliminating hallucinations while preserving the original tone. Nextdoor then introduced a rejection sampling mechanism featuring a dedicated reward model that predicts user engagement and accepts generated subjects only when their score exceeds the user-written baseline. This combined system lifted sessions by 1%, boosted Weekly Active Users by 0.4%, and increased advertising revenue by 1%."
---

# Let AI Entertain You: Increasing User Engagement with Generative AI and Rejection Sampling

[Nextdoor](https://yomu.fyi/company/nextdoor) · Jaewon Yang · Oct 16, 2023

**Type:** Problem & solution

## Summary

Nextdoor implemented a generative AI framework to improve email notification subject lines for its New and Trending emails. Using off-the-shelf ChatGPT API outputs directly resulted in marketing-like phrases, hallucinations, and lower click-through rates compared to original user-generated post snippets. To resolve this, the team prompted the language model strictly to extract authentic post phrases without rewriting, eliminating hallucinations while preserving the original tone. Nextdoor then introduced a rejection sampling mechanism featuring a dedicated reward model that predicts user engagement and accepts generated subjects only when their score exceeds the user-written baseline. This combined system lifted sessions by 1%, boosted Weekly Active Users by 0.4%, and increased advertising revenue by 1%.

## Context

Nextdoor historically populated New and Trending notification email subject lines with the first few words of a post, which often contained uninformative greetings. Initial attempts to generate subject lines using the vanilla ChatGPT API produced marketing-like phrasing, suffered from hallucinations, and generated only 56% of the clicks achieved by user-written subject lines.

## Approach / What changed

Nextdoor implemented a rejection sampling system using two AI models. The subject line generator uses OpenAI API prompt engineering to extract interesting phrases verbatim without rewriting. A separate reward model evaluates engagement and accepts the generated subject only if its score beats the original user-written subject line. Caching reduced serving costs to 1/600 compared to brute force, while daily monitoring tracks reward model prediction accuracy.

## Takeaways

- Prompting the language model to extract verbatim post phrases rather than rewrite them eliminates hallucinations, avoids spammy marketing phrasing, and improved sessions by 3% relative to rewriting.
- Prompt engineering alone encountered a performance ceiling that failed to beat the control, making a reward model essential for selecting winning subject lines.
- The final rejection sampling system achieved a 1% lift in user sessions, a 0.4% increase in Weekly Active Users, and a 1% increase in Ads revenue in A/B testing.

**Tags:** [Caching](https://yomu.fyi/topic/caching), [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning)

- Source: [Nextdoor](https://engblog.nextdoor.com/let-ai-entertain-you-increasing-user-engagement-with-generative-ai-and-rejection-sampling-50a402264f56)
- Source URL: https://engblog.nextdoor.com/let-ai-entertain-you-increasing-user-engagement-with-generative-ai-and-rejection-sampling-50a402264f56
- Ingested by Yomu: 2026-08-27T14:57:31.104Z

[Read original post](https://engblog.nextdoor.com/let-ai-entertain-you-increasing-user-engagement-with-generative-ai-and-rejection-sampling-50a402264f56)