How we optimized Dash's relevance judge with DSPy

Summary

Dropbox Dash optimized its LLM-as-a-judge relevance scoring system using DSPy to migrate from expensive proprietary models to cheaper open-weight alternatives. By establishing automated feedback loops based on human agreement and strict JSON format validation, the team systematically generated robust prompts for new models. This reduced human-score disagreement by 45% and slashed model adaptation time from weeks to days while enabling 10x to 100x more data labeling.

Takeaways

1. 1

   Using DSPy with the GEPA reflection optimizer cut model adaptation time from up to two weeks of manual prompt engineering to just one to two days.

2. 2

   Automated prompt optimization loops require guardrails to prevent the optimizer from overfitting to specific dataset keywords or mutating core task definitions like rating scales.

3. 3

   Optimizing against structural reliability as part of the objective function reduced malformed JSON responses in smaller models (like Gemma-3-12B) by more than 97%.