How we used DSPy to turn AI evaluations into better responses in Dash chat

Summary

Dropbox improved its Dash chat agent by establishing an automated optimization loop powered by DSPy and LLM-as-judge evaluations. Engineers first calibrated their LLM judges against human-annotated interaction traces, then used those judges to systematically optimize the agent's system prompts via offline counterfactual replay. This automated workflow reduced incomplete responses by 26% while decreasing overall token consumption.

Takeaways

1. 1

   Agent evaluation requires scoring multi-step execution trajectories—including intent recognition, tool selection, and grounding—using structured failure codes and reasoning notes rather than just judging final outputs.

2. 2

   A two-stage DSPy optimization pipeline can first calibrate LLM judges against human ratings (using algorithms like GEPA and MIPROv2) and then use those calibrated judges to optimize the agent's prompt.

3. 3

   Automating prompt optimization with offline counterfactual replays doubled experimentation velocity, reduced incomplete answers by 26%, and cut overall token usage by 5.4%.