Using LLMs to amplify human labeling and improve Dash search relevance

Summary

Dropbox Dash uses large language models (LLMs) to amplify human labeling efforts for training its search relevance and ranking models. By validating and optimizing LLM evaluators against a small set of human-labeled internal data, Dropbox creates massive, high-quality training datasets offline for production rankers like XGBoost without incurring high latency or latency costs at query time.

Takeaways

1. 1

   Offline LLM evaluators act as teacher models that generate millions of relevance labels to train fast, lightweight ranking models like XGBoost.

2. 2

   Providing LLMs with tool access helps resolve company-specific acronyms and context that standard query-document pairs lack.

3. 3

   Prompt optimization frameworks like DSPy and discrepancy sampling (comparing user clicks/skips to LLM scores) efficiently drive down evaluation error against human baselines.