---
title: "How Etsy Uses LLMs to Improve Search Relevance"
description: "Etsy’s Search Relevance team addresses a limitation of engagement-based search signals: popular listings can attract clicks even when they are not the best match for a query. It introduces semantic relevance, measured through a framework combining human-curated golden labels, an LLM annotator, cascaded models, and production integrations, with categories for relevant, partially relevant, and irrelevant query-listing pairs. The data pipeline uses sampled search-log pairs, dual human annotation, an evolving guideline, few-shot chain-of-thought prompting with o3 in LangGraph, and self-consistency sampling; a Qwen 3 VL 4B teacher then labels data at scale before distilling into a BERT-based two-tower student. The student supports real-time use with less than 10ms additional latency, while the framework evaluates and enhances search relevance alongside engagement signals."
---

# How Etsy Uses LLMs to Improve Search Relevance

[Etsy](https://yomu.fyi/company/etsy) · Yuqing Zhang · Jan 16, 2026

**Type:** Explainer

## Summary

Etsy’s Search Relevance team addresses a limitation of engagement-based search signals: popular listings can attract clicks even when they are not the best match for a query. It introduces semantic relevance, measured through a framework combining human-curated golden labels, an LLM annotator, cascaded models, and production integrations, with categories for relevant, partially relevant, and irrelevant query-listing pairs. The data pipeline uses sampled search-log pairs, dual human annotation, an evolving guideline, few-shot chain-of-thought prompting with o3 in LangGraph, and self-consistency sampling; a Qwen 3 VL 4B teacher then labels data at scale before distilling into a BERT-based two-tower student. The student supports real-time use with less than 10ms additional latency, while the framework evaluates and enhances search relevance alongside engagement signals.

## Context

Etsy’s historical reliance on clicks, add-to-carts, and purchases as relevance proxies can favor popular listings over those that best match a specific query. Large-scale human annotation is also time-consuming and expensive, while off-the-shelf and larger LLMs present domain-shift, accuracy, latency, and cost trade-offs.

## Approach / What changed

The framework anchors relevance judgments in human-curated golden labels, uses an o3-based LLM annotator with few-shot chain-of-thought prompting and self-consistency sampling, and scales annotation through a Qwen 3 VL 4B teacher model. A BERT-based two-tower student is distilled for real-time inference, and relevance signals are integrated into Etsy’s search systems.

## Takeaways

- Human-curated golden labels define relevance categories and validate the LLM annotator; the LLM scales human-aligned judgments rather than replacing human judgment.
- A three-tier cascade uses an LLM annotator for accuracy, a Qwen 3 VL 4B teacher for high-throughput annotation, and a lightweight BERT-based two-tower student for production inference.
- The distilled student adds less than 10ms of latency, enabling semantic relevance to complement engagement signals in real-time search and potentially improve visibility for small or new sellers.

**Tags:** [LangGraph](https://yomu.fyi/topic/langgraph), [LLMs](https://yomu.fyi/topic/llm), [Model Evaluation](https://yomu.fyi/topic/model-evaluation), [Search](https://yomu.fyi/topic/search)

- Source: [Etsy](https://www.etsy.com/codeascraft/how-etsy-uses-llms-to-improve-search-relevance)
- Source URL: https://www.etsy.com/codeascraft/how-etsy-uses-llms-to-improve-search-relevance
- Ingested by Yomu: 2026-09-09T20:02:15.852Z

[Read original post](https://www.etsy.com/codeascraft/how-etsy-uses-llms-to-improve-search-relevance)