# How we optimized Dash's relevance judge with DSPy

[Dropbox](https://yomu.fyi/company/dropbox) · Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura,Mike White,Kevin Altschuler,Facundo Agriel,Ishan Mishra,Eric Wang,Dmitriy Meyerzon · Mar 17, 2026

## Summary

Dropbox Dash optimized its LLM-as-a-judge relevance scoring system using DSPy to migrate from expensive proprietary models to cheaper open-weight alternatives. By establishing automated feedback loops based on human agreement and strict JSON format validation, the team systematically generated robust prompts for new models. This reduced human-score disagreement by 45% and slashed model adaptation time from weeks to days while enabling 10x to 100x more data labeling.

## Takeaways

- Using DSPy with the GEPA reflection optimizer cut model adaptation time from up to two weeks of manual prompt engineering to just one to two days.
- Automated prompt optimization loops require guardrails to prevent the optimizer from overfitting to specific dataset keywords or mutating core task definitions like rating scales.
- Optimizing against structural reliability as part of the objective function reduced malformed JSON responses in smaller models (like Gemma-3-12B) by more than 97%.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance), [Search](https://yomu.fyi/topic/search)

[Read original post](https://dropbox.tech/machine-learning/optimizing-dropbox-dash-relevance-judge-with-dspy)