How we optimized Dash's relevance judge with DSPy

Summary

Dropbox Dash optimized its LLM-as-a-judge relevance scoring system using DSPy to migrate from expensive proprietary models to cheaper open-weight alternatives. By establishing automated feedback loops based on human agreement and strict JSON format validation, the team systematically generated robust prompts for new models. This reduced human-score disagreement by 45% and slashed model adaptation time from weeks to days while enabling 10x to 100x more data labeling.

Takeaways

• Using DSPy with the GEPA reflection optimizer cut model adaptation time from up to two weeks of manual prompt engineering to just one to two days.
• Automated prompt optimization loops require guardrails to prevent the optimizer from overfitting to specific dataset keywords or mutating core task definitions like rating scales.
• Optimizing against structural reliability as part of the objective function reduced malformed JSON responses in smaller models (like Gemma-3-12B) by more than 97%.

Related reading

• Dropbox · Jun 25, 2026

  How we used DSPy to turn AI evaluations into better responses in Dash chat

  Dropbox improved its Dash chat agent by establishing an automated optimization loop powered by DSPy and LLM-as-judge evaluations. Engineers first calibrated their LLM judges against human-annotated interaction traces, then used those judges to systematically optimize the agent's system prompts via offline counterfactual replay. This automated workflow reduced incomplete responses by 26% while decreasing overall token consumption.

  Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon

• Dropbox · Feb 26, 2026

  Using LLMs to amplify human labeling and improve Dash search relevance

  Dropbox Dash uses large language models (LLMs) to amplify human labeling efforts for training its search relevance and ranking models. By validating and optimizing LLM evaluators against a small set of human-labeled internal data, Dropbox creates massive, high-quality training datasets offline for production rankers like XGBoost without incurring high latency or latency costs at query time.

  Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura,Mike White,Kevin Altschuler,Facundo Agriel,Ishan Mishra,Eric Wang,Dmitriy Meyerzon,Dmitriy Meyerzon

• Dropbox · Mar 25, 2026

  Reducing our monorepo size to improve developer velocity

  Dropbox reduced its core server monorepo size from 87GB to 20GB after approaching GitHub's 100GB limit and suffering from hour-long clone times. The excessive bloat was traced to Git's delta compression heuristics interacting poorly with internationalization directory paths, generating inefficient diffs across different languages. By collaborating with GitHub to run an aggressive server-side repack with tuned window and depth settings, Dropbox cut clone times to under 15 minutes.

  Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura,Mike White,Kevin Altschuler,Facundo Agriel,Ishan Mishra

• Dropbox · Jul 20, 2026

  How our universal content processing platform Riviera evolved for AI and beyond

  Dropbox developed Riviera, a centralized content processing platform that evolved from an internal file preview generator into a shared infrastructure supporting over 300 file formats. By decomposing complex conversion tasks into modular, reusable transformations, the system now powers search indexing, media transcoding, and AI context extraction across Dropbox products.

  Ilya Yakovlev,Andrew Cheung,Binoy Dash