coSTAR: How we ship AI agents at Databricks fast, without breaking things

Summary

Databricks describes coSTAR, a methodology for testing and iteratively improving long-running, stochastic AI agents without relying on manual review for every change. It addresses nondeterministic outputs, slow executions, cascading failures, and subjective quality by treating scenarios as fixtures, MLflow traces as a reusable harness, and judges as an automated test suite. Its two coupled STAR loops align judges with human experts using a Golden Set, then use those judges to refine the agent until it passes the scenarios. Agentic judges selectively explore traces with tools instead of receiving entire, potentially thousands-line executions, while persisted traces allow rescoring without rerunning agents. The approach can also reuse judges for production monitoring, but remains limited by manually authored scenarios, judge-calibration costs, coverage gaps, overfitting, and difficult root-cause attribution.

Takeaways

1. 1

   Persisting MLflow traces separates agent execution from scoring, allowing thresholds or judges to change and be evaluated without rerunning expensive scenarios.

2. 2

   Agentic judges use targeted trace-exploration tools to assess properties such as skill invocation, documented best practices, and output assets instead of consuming an entire trace at once.

3. 3

   coSTAR can improve only within its scenario and judge coverage; manually authored scenarios, judge recalibration, overfitting, and unclear multi-step root causes remain limitations.

FAQ