# Beyond the Menu Tree: How Yelp Built a Smarter Customer Success Chatbot with AI

[Yelp](https://yomu.fyi/company/yelp) · Lina Lee, Machine Learning Engineer; Nelson Lee, Engineering Manager · May 27, 2026

**Type:** Problem & solution

## Summary

Yelp transitioned from a legacy customer support chatbot that relied on rigid phrase matching and static menus to a retrieval-augmented generation system. The updated architecture classifies incoming queries using a large language model to route users into one of five specialized workflows, including QA, Billing, Refund, Cancel, and Review. For general QA, the system uses FAISS to search an in-memory vectorstore built exclusively from article metadata such as titles, summaries, and headers rather than raw article text. Retrieved articles are supplied to the language model as prompt context, followed by automated validations for safety, character limits, and hallucinated hyperlinks. In production A/B testing, this metadata-driven design doubled the chatbot resolution rate while achieving approximately 94% recall@5.

## Context

Yelp's legacy customer support chatbot relied on a 2-step menu tree and rigid matching against a fixed set of phrases, causing failures whenever user inquiries diverged from exact expected phrases or menu structures.

## Approach / What changed

Yelp built an LLM-assisted chatbot that routes queries across five specialized workflows (QA, Billing, Refund, Cancel, Review) based on frequency and legal, financial, or churn risk. The QA workflow uses RAG backed by an 8-megabyte in-memory FAISS vectorstore of metadata embeddings (titles, summaries, headers) generated via text-embedding-ada-002 from approximately 370 Support Center articles. An automated daily batch job exports article data to AWS S3, which container instances download and index on startup. Prompts pass up to five whole matching articles to an LLM, accompanied by output validation checks for trust and safety, length limits, and verified URLs.

## Takeaways

- Embedding individual metadata segments like titles and headers avoided signal dilution and outperformed embedding whole articles or smaller paragraph chunks.
- An in-memory FAISS vectorstore built from article metadata reached ~94% recall@5 while occupying only about 8 megabytes for roughly 370 articles.
- A/B testing revealed that moving from the static phrase-matching bot to the RAG-assisted chatbot doubled the customer support resolution rate.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [AWS](https://yomu.fyi/topic/aws), [LLMs](https://yomu.fyi/topic/llm), [Search](https://yomu.fyi/topic/search)

[Read original post](https://engineeringblog.yelp.com/2026/05/beyond-menu-tree.html)