Let AI Entertain You: Increasing User Engagement with Generative AI and Rejection Sampling

Summary

Nextdoor implemented a generative AI framework to improve email notification subject lines for its New and Trending emails. Using off-the-shelf ChatGPT API outputs directly resulted in marketing-like phrases, hallucinations, and lower click-through rates compared to original user-generated post snippets. To resolve this, the team prompted the language model strictly to extract authentic post phrases without rewriting, eliminating hallucinations while preserving the original tone. Nextdoor then introduced a rejection sampling mechanism featuring a dedicated reward model that predicts user engagement and accepts generated subjects only when their score exceeds the user-written baseline. This combined system lifted sessions by 1%, boosted Weekly Active Users by 0.4%, and increased advertising revenue by 1%.

Context

Nextdoor historically populated New and Trending notification email subject lines with the first few words of a post, which often contained uninformative greetings. Initial attempts to generate subject lines using the vanilla ChatGPT API produced marketing-like phrasing, suffered from hallucinations, and generated only 56% of the clicks achieved by user-written subject lines.

Approach / What changed

Nextdoor implemented a rejection sampling system using two AI models. The subject line generator uses OpenAI API prompt engineering to extract interesting phrases verbatim without rewriting. A separate reward model evaluates engagement and accepts the generated subject only if its score beats the original user-written subject line. Caching reduced serving costs to 1/600 compared to brute force, while daily monitoring tracks reward model prediction accuracy.

Takeaways

• Prompting the language model to extract verbatim post phrases rather than rewrite them eliminates hallucinations, avoids spammy marketing phrasing, and improved sessions by 3% relative to rewriting.
• Prompt engineering alone encountered a performance ceiling that failed to beat the control, making a reward model essential for selecting winning subject lines.
• The final rejection sampling system achieved a 1% lift in user sessions, a 0.4% increase in Weekly Active Users, and a 1% increase in Ads revenue in A/B testing.

Related reading