Understanding Etsy’s Vast Inventory with LLMs

Summary

Etsy’s seller-created inventory spans more than 100 million items and thousands of categories, making it difficult to capture attributes such as color, material, or niche properties without global SKUs. Because sellers often provide descriptions, titles, and photos instead of complete structured fields, Etsy uses LLMs to distill unstructured listing data into attributes for search filters, comparisons, and other experiences. The pipeline combines seller data, expert-selected few-shot examples, taxonomy logic, and category rules; parallel prompts are routed through LiteLLM, then responses are validated with Pydantic dataclasses and monitored with tracing, metrics, alerts, and performance checks. Human-reviewed silver labels support evaluation, while target categories increased complete attribute coverage from 31% to 91%, and inferred attributes improved search-filter engagement and overall post-click conversion.

Takeaways

1. 1

   Human-annotated datasets were evaluated with precision, recall, and Jaccard index, but annotator errors, time, and cost led Etsy to use LLM-generated silver labels reviewed by domain experts.

2. 2

   Each listing’s JSON context combines seller data, few-shot examples, product-taxonomy logic, and category rules; parallel LLM responses are parsed into Pydantic dataclasses for type and business-rule validation.

3. 3

   Inference errors are logged and surfaced through metrics and alerts, while sampled traces support debugging and sample-based performance checks can terminate the pipeline when metrics deviate significantly from baseline.