Using LLMs to amplify human labeling and improve Dash search relevance

Summary

Dropbox Dash uses large language models (LLMs) to amplify human labeling efforts for training its search relevance and ranking models. By validating and optimizing LLM evaluators against a small set of human-labeled internal data, Dropbox creates massive, high-quality training datasets offline for production rankers like XGBoost without incurring high latency or latency costs at query time.

Takeaways

• Offline LLM evaluators act as teacher models that generate millions of relevance labels to train fast, lightweight ranking models like XGBoost.
• Providing LLMs with tool access helps resolve company-specific acronyms and context that standard query-document pairs lack.
• Prompt optimization frameworks like DSPy and discrepancy sampling (comparing user clicks/skips to LLM scores) efficiently drive down evaluation error against human baselines.

Related reading

• Dropbox · Mar 17, 2026

  How we optimized Dash's relevance judge with DSPy

  Dropbox Dash optimized its LLM-as-a-judge relevance scoring system using DSPy to migrate from expensive proprietary models to cheaper open-weight alternatives. By establishing automated feedback loops based on human agreement and strict JSON format validation, the team systematically generated robust prompts for new models. This reduced human-score disagreement by 45% and slashed model adaptation time from weeks to days while enabling 10x to 100x more data labeling.

  Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura,Mike White,Kevin Altschuler,Facundo Agriel,Ishan Mishra,Eric Wang,Dmitriy Meyerzon

• Dropbox · Jul 20, 2026

  How our universal content processing platform Riviera evolved for AI and beyond

  Dropbox developed Riviera, a centralized content processing platform that evolved from an internal file preview generator into a shared infrastructure supporting over 300 file formats. By decomposing complex conversion tasks into modular, reusable transformations, the system now powers search indexing, media transcoding, and AI context extraction across Dropbox products.

  Ilya Yakovlev,Andrew Cheung,Binoy Dash

• Dropbox · Jun 25, 2026

  How we used DSPy to turn AI evaluations into better responses in Dash chat

  Dropbox improved its Dash chat agent by establishing an automated optimization loop powered by DSPy and LLM-as-judge evaluations. Engineers first calibrated their LLM judges against human-annotated interaction traces, then used those judges to systematically optimize the agent's system prompts via offline counterfactual replay. This automated workflow reduced incomplete responses by 26% while decreasing overall token consumption.

  Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon

• Dropbox · Mar 25, 2026

  Reducing our monorepo size to improve developer velocity

  Dropbox reduced its core server monorepo size from 87GB to 20GB after approaching GitHub's 100GB limit and suffering from hour-long clone times. The excessive bloat was traced to Git's delta compression heuristics interacting poorly with internationalization directory paths, generating inefficient diffs across different languages. By collaborating with GitHub to run an aggressive server-side repack with tuned window and depth settings, Dropbox cut clone times to under 15 minutes.

  Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura,Mike White,Kevin Altschuler,Facundo Agriel,Ishan Mishra