Loading post…
NextdoorJaewon YangProblem & solution
Let AI Entertain You: Increasing User Engagement with Generative AI and Rejection Sampling
Summary
Nextdoor implemented a generative AI framework to improve email notification subject lines for its New and Trending emails. Using off-the-shelf ChatGPT API outputs directly resulted in marketing-like phrases, hallucinations, and lower click-through rates compared to original user-generated post snippets. To resolve this, the team prompted the language model strictly to extract authentic post phrases without rewriting, eliminating hallucinations while preserving the original tone. Nextdoor then introduced a rejection sampling mechanism featuring a dedicated reward model that predicts user engagement and accepts generated subjects only when their score exceeds the user-written baseline. This combined system lifted sessions by 1%, boosted Weekly Active Users by 0.4%, and increased advertising revenue by 1%.
Context
Nextdoor historically populated New and Trending notification email subject lines with the first few words of a post, which often contained uninformative greetings. Initial attempts to generate subject lines using the vanilla ChatGPT API produced marketing-like phrasing, suffered from hallucinations, and generated only 56% of the clicks achieved by user-written subject lines.
Approach / What changed
Nextdoor implemented a rejection sampling system using two AI models. The subject line generator uses OpenAI API prompt engineering to extract interesting phrases verbatim without rewriting. A separate reward model evaluates engagement and accepts the generated subject only if its score beats the original user-written subject line. Caching reduced serving costs to 1/600 compared to brute force, while daily monitoring tracks reward model prediction accuracy.
Takeaways
Prompting the language model to extract verbatim post phrases rather than rewrite them eliminates hallucinations, avoids spammy marketing phrasing, and improved sessions by 3% relative to rewriting.
Prompt engineering alone encountered a performance ceiling that failed to beat the control, making a reward model essential for selecting winning subject lines.
The final rejection sampling system achieved a 1% lift in user sessions, a 0.4% increase in Weekly Active Users, and a 1% increase in Ads revenue in A/B testing.
Source and freshness
- Source
- engblog.nextdoor.com
- Published
- Added to Yomu
- Last updated
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.