How Etsy Uses LLMs to Improve Search Relevance

Summary

Etsy’s Search Relevance team addresses a limitation of engagement-based search signals: popular listings can attract clicks even when they are not the best match for a query. It introduces semantic relevance, measured through a framework combining human-curated golden labels, an LLM annotator, cascaded models, and production integrations, with categories for relevant, partially relevant, and irrelevant query-listing pairs. The data pipeline uses sampled search-log pairs, dual human annotation, an evolving guideline, few-shot chain-of-thought prompting with o3 in LangGraph, and self-consistency sampling; a Qwen 3 VL 4B teacher then labels data at scale before distilling into a BERT-based two-tower student. The student supports real-time use with less than 10ms additional latency, while the framework evaluates and enhances search relevance alongside engagement signals.

Takeaways

1. 1

   Human-curated golden labels define relevance categories and validate the LLM annotator; the LLM scales human-aligned judgments rather than replacing human judgment.

2. 2

   A three-tier cascade uses an LLM annotator for accuracy, a Qwen 3 VL 4B teacher for high-throughput annotation, and a lightweight BERT-based two-tower student for production inference.

3. 3

   The distilled student adds less than 10ms of latency, enabling semantic relevance to complement engagement signals in real-time search and potentially improve visibility for small or new sellers.