Building Menu Vision: Real-Time Dish Recognition

Summary

Yelp developed Menu Vision to help diners recognize menu items and view corresponding user-generated photos and reviews in real time using their phone cameras. The feature evolved from a two-day hackathon Android prototype into a production system deployed across iOS and Android. Client devices perform all optical character recognition, text cleanup, and dish matching locally using native machine learning frameworks, while dish metadata is prefetched upon opening a restaurant page. Backend pipelines combine partner data, reviews, and photo captions into a Cassandra database, filtering exclusively for dishes with photos. To handle optical character recognition errors and name variations, the matching pipeline runs exact matching, bidirectional substring matching, and Jaro-Winkler similarity matching.

Context

Restaurant diners reading text menus struggle to understand portion sizes or diner satisfaction without manually searching for individual dishes, which is cumbersome while seated at a table.

Approach / What changed

Yelp built Menu Vision to identify dishes using on-device optical character recognition and native machine learning frameworks. The backend aggregates menu data from partners, reviews, and photo captions, storing deduplicated items with photos in Cassandra for fast retrieval. Client applications prefetch dish data upon viewing a restaurant page and execute a three-phase matching pipeline consisting of exact matching, bidirectional substring matching, and Jaro-Winkler similarity matching. The interface renders dish cards with photos, review counts, and prices over live camera views, alongside QR code detection and fallback inventory lists.

Takeaways

• Executing text recognition, cleanup, and matching on-device with native frameworks eliminates the latency and network dependency of transmitting camera images to servers.
• Prefetching restaurant dish data upon page load ensures necessary records exist locally before the user activates the camera scanner.
• A three-phase matching strategy using exact match, bidirectional substring search, and Jaro-Winkler similarity resolves OCR noise, modifiers, and spelling discrepancies.

Related reading

• Yelp · Mar 27, 2026

  Building Biz Ask Anything: From Prototype to Product

  Yelp expanded Yelp Assistant to business pages by developing a question-answering architecture that extracts concise, evidence-backed answers from reviews, photos, and structured facts. The production system replaces static snapshots with near-real-time streaming pipelines for reviews and structured properties alongside weekly batch ingestion for menus and website data. For each incoming query, the system evaluates intent, retrieves relevant context through keyword and embedding searches, and constructs targeted prompts with dynamically selected few-shot examples. Synthesized answers are augmented with supporting visuals and citations before being streamed token-by-token to users. Generating suggested questions directly from specific business content rather than generic categories improved user engagement by roughly fifty percent and reduced unanswerable queries by twenty-six percent.

  Maria Christoforaki, Group Tech Lead; Shree Shalini Pusapati, Software Engineer

• Yelp · May 27, 2026