# How we used DSPy to turn AI evaluations into better responses in Dash chat

[Dropbox](https://yomu.fyi/company/dropbox) · Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon · Jun 25, 2026

## Summary

Dropbox improved its Dash chat agent by establishing an automated optimization loop powered by DSPy and LLM-as-judge evaluations. Engineers first calibrated their LLM judges against human-annotated interaction traces, then used those judges to systematically optimize the agent's system prompts via offline counterfactual replay. This automated workflow reduced incomplete responses by 26% while decreasing overall token consumption.

## Takeaways

- Agent evaluation requires scoring multi-step execution trajectories—including intent recognition, tool selection, and grounding—using structured failure codes and reasoning notes rather than just judging final outputs.
- A two-stage DSPy optimization pipeline can first calibrate LLM judges against human ratings (using algorithms like GEPA and MIPROv2) and then use those calibrated judges to optimize the agent's prompt.
- Automating prompt optimization with offline counterfactual replays doubled experimentation velocity, reduced incomplete answers by 26%, and cut overall token usage by 5.4%.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Testing](https://yomu.fyi/topic/testing)

[Read original post](https://dropbox.tech/machine-learning/how-we-turned-ai-evaluations-into-better-responses-in-dash-chat)