Beyond the Menu Tree: How Yelp Built a Smarter Customer Success Chatbot with AI

Summary

Yelp transitioned from a legacy customer support chatbot that relied on rigid phrase matching and static menus to a retrieval-augmented generation system. The updated architecture classifies incoming queries using a large language model to route users into one of five specialized workflows, including QA, Billing, Refund, Cancel, and Review. For general QA, the system uses FAISS to search an in-memory vectorstore built exclusively from article metadata such as titles, summaries, and headers rather than raw article text. Retrieved articles are supplied to the language model as prompt context, followed by automated validations for safety, character limits, and hallucinated hyperlinks. In production A/B testing, this metadata-driven design doubled the chatbot resolution rate while achieving approximately 94% recall@5.

Context

Yelp's legacy customer support chatbot relied on a 2-step menu tree and rigid matching against a fixed set of phrases, causing failures whenever user inquiries diverged from exact expected phrases or menu structures.

Approach / What changed

Yelp built an LLM-assisted chatbot that routes queries across five specialized workflows (QA, Billing, Refund, Cancel, Review) based on frequency and legal, financial, or churn risk. The QA workflow uses RAG backed by an 8-megabyte in-memory FAISS vectorstore of metadata embeddings (titles, summaries, headers) generated via text-embedding-ada-002 from approximately 370 Support Center articles. An automated daily batch job exports article data to AWS S3, which container instances download and index on startup. Prompts pass up to five whole matching articles to an LLM, accompanied by output validation checks for trust and safety, length limits, and verified URLs.

Takeaways

• Embedding individual metadata segments like titles and headers avoided signal dilution and outperformed embedding whole articles or smaller paragraph chunks.
• An in-memory FAISS vectorstore built from article metadata reached ~94% recall@5 while occupying only about 8 megabytes for roughly 370 articles.
• A/B testing revealed that moving from the static phrase-matching bot to the RAG-assisted chatbot doubled the customer support resolution rate.

Related reading

• Yelp · Aug 20, 2026

  Building Menu Vision: Real-Time Dish Recognition

  Yelp developed Menu Vision to help diners recognize menu items and view corresponding user-generated photos and reviews in real time using their phone cameras. The feature evolved from a two-day hackathon Android prototype into a production system deployed across iOS and Android. Client devices perform all optical character recognition, text cleanup, and dish matching locally using native machine learning frameworks, while dish metadata is prefetched upon opening a restaurant page. Backend pipelines combine partner data, reviews, and photo captions into a Cassandra database, filtering exclusively for dishes with photos. To handle optical character recognition errors and name variations, the matching pipeline runs exact matching, bidirectional substring matching, and Jaro-Winkler similarity matching.

  Arpitha Dudi, Growth Tech Lead

• Yelp · Mar 27, 2026

  Building Biz Ask Anything: From Prototype to Product