---
title: "coSTAR: How we ship AI agents at Databricks fast, without breaking things"
description: "Databricks describes coSTAR, a methodology for testing and iteratively improving long-running, stochastic AI agents without relying on manual review for every change. It addresses nondeterministic outputs, slow executions, cascading failures, and subjective quality by treating scenarios as fixtures, MLflow traces as a reusable harness, and judges as an automated test suite. Its two coupled STAR loops align judges with human experts using a Golden Set, then use those judges to refine the agent until it passes the scenarios. Agentic judges selectively explore traces with tools instead of receiving entire, potentially thousands-line executions, while persisted traces allow rescoring without rerunning agents. The approach can also reuse judges for production monitoring, but remains limited by manually authored scenarios, judge-calibration costs, coverage gaps, overfitting, and difficult root-cause attribution."
---

# coSTAR: How we ship AI agents at Databricks fast, without breaking things

[Databricks](https://yomu.fyi/company/databricks) · Alkis Polyzotis · Sep 18, 2026

**Type:** Explainer

## Summary

Databricks describes coSTAR, a methodology for testing and iteratively improving long-running, stochastic AI agents without relying on manual review for every change. It addresses nondeterministic outputs, slow executions, cascading failures, and subjective quality by treating scenarios as fixtures, MLflow traces as a reusable harness, and judges as an automated test suite. Its two coupled STAR loops align judges with human experts using a Golden Set, then use those judges to refine the agent until it passes the scenarios. Agentic judges selectively explore traces with tools instead of receiving entire, potentially thousands-line executions, while persisted traces allow rescoring without rerunning agents. The approach can also reuse judges for production monitoring, but remains limited by manually authored scenarios, judge-calibration costs, coverage gaps, overfitting, and difficult root-cause attribution.

## Context

Databricks had agents performing long-running tasks and generating code and data or AI assets, but its earlier workflow depended on manually reviewing outputs and lacked a comprehensive automated test suite. Agent testing is complicated by nondeterminism, expensive executions, cascading failures, and subjective quality criteria.

## Approach / What changed

coSTAR uses two coupled Scenario, Trace, Assess, Refine loops. Scenario definitions specify initial state, prompts, and expected outcomes; a harness runs the agent under test and persists MLflow traces; agentic judges inspect traces with targeted tools; and a coding assistant refines the implementation. A Golden Set of human-graded examples calibrates the judges before they drive automated refinement.

## Takeaways

- Persisting MLflow traces separates agent execution from scoring, allowing thresholds or judges to change and be evaluated without rerunning expensive scenarios.
- Agentic judges use targeted trace-exploration tools to assess properties such as skill invocation, documented best practices, and output assets instead of consuming an entire trace at once.
- coSTAR can improve only within its scenario and judge coverage; manually authored scenarios, judge recalibration, overfitting, and unclear multi-step root causes remain limitations.

## FAQ

### Why does coSTAR use agentic judges instead of passing a full trace to an LLM?

A single trace can contain thousands of lines, and placing it all in a judge’s context is described as expensive, fragile, and harmful to judgment quality. Agentic judges selectively explore traces with tools and check targeted properties.

### How are the judges checked before they drive agent refinement?

Domain experts manually grade examples in a Golden Set. The judge loop refines judges until their assessments align with those human evaluations.

### Can the same evaluations be used after an agent reaches production?

Yes. The post says the same judges used during evaluation can monitor live traffic, so the evaluation investment can also support production monitoring.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [MLflow](https://yomu.fyi/topic/mlflow), [Model Evaluation](https://yomu.fyi/topic/model-evaluation), [Testing](https://yomu.fyi/topic/testing)

- Source: [Databricks](https://www.databricks.com/blog/costar-how-we-ship-ai-agents-databricks-fast-without-breaking-things)
- Source URL: https://www.databricks.com/blog/costar-how-we-ship-ai-agents-databricks-fast-without-breaking-things
- Ingested by Yomu: 2026-09-17T14:13:10.381Z

[Read original post](https://www.databricks.com/blog/costar-how-we-ship-ai-agents-databricks-fast-without-breaking-things)