# Building Menu Vision: Real-Time Dish Recognition

[Yelp](https://yomu.fyi/company/yelp) · Arpitha Dudi, Growth Tech Lead · Aug 20, 2026

**Type:** Problem & solution

## Summary

Yelp developed Menu Vision to help diners recognize menu items and view corresponding user-generated photos and reviews in real time using their phone cameras. The feature evolved from a two-day hackathon Android prototype into a production system deployed across iOS and Android. Client devices perform all optical character recognition, text cleanup, and dish matching locally using native machine learning frameworks, while dish metadata is prefetched upon opening a restaurant page. Backend pipelines combine partner data, reviews, and photo captions into a Cassandra database, filtering exclusively for dishes with photos. To handle optical character recognition errors and name variations, the matching pipeline runs exact matching, bidirectional substring matching, and Jaro-Winkler similarity matching.

## Context

Restaurant diners reading text menus struggle to understand portion sizes or diner satisfaction without manually searching for individual dishes, which is cumbersome while seated at a table.

## Approach / What changed

Yelp built Menu Vision to identify dishes using on-device optical character recognition and native machine learning frameworks. The backend aggregates menu data from partners, reviews, and photo captions, storing deduplicated items with photos in Cassandra for fast retrieval. Client applications prefetch dish data upon viewing a restaurant page and execute a three-phase matching pipeline consisting of exact matching, bidirectional substring matching, and Jaro-Winkler similarity matching. The interface renders dish cards with photos, review counts, and prices over live camera views, alongside QR code detection and fallback inventory lists.

## Takeaways

- Executing text recognition, cleanup, and matching on-device with native frameworks eliminates the latency and network dependency of transmitting camera images to servers.
- Prefetching restaurant dish data upon page load ensures necessary records exist locally before the user activates the camera scanner.
- A three-phase matching strategy using exact match, bidirectional substring search, and Jaro-Winkler similarity resolves OCR noise, modifiers, and spelling discrepancies.

**Tags:** [Android](https://yomu.fyi/topic/android), [iOS](https://yomu.fyi/topic/ios), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance), [Search](https://yomu.fyi/topic/search)

[Read original post](https://engineeringblog.yelp.com/2026/08/building-menu-vision-real-time-dish-recognition.html)