Beyond the Menu Tree: How Yelp Built a Smarter Customer Success Chatbot with AI

Summary

Yelp transitioned from a legacy customer support chatbot that relied on rigid phrase matching and static menus to a retrieval-augmented generation system. The updated architecture classifies incoming queries using a large language model to route users into one of five specialized workflows, including QA, Billing, Refund, Cancel, and Review. For general QA, the system uses FAISS to search an in-memory vectorstore built exclusively from article metadata such as titles, summaries, and headers rather than raw article text. Retrieved articles are supplied to the language model as prompt context, followed by automated validations for safety, character limits, and hallucinated hyperlinks. In production A/B testing, this metadata-driven design doubled the chatbot resolution rate while achieving approximately 94% recall@5.

Takeaways

1. 1

   Embedding individual metadata segments like titles and headers avoided signal dilution and outperformed embedding whole articles or smaller paragraph chunks.

2. 2

   An in-memory FAISS vectorstore built from article metadata reached ~94% recall@5 while occupying only about 8 megabytes for roughly 370 articles.

3. 3

   A/B testing revealed that moving from the static phrase-matching bot to the RAG-assisted chatbot doubled the customer support resolution rate.