---
title: "Hill-Climbing Agent Quality with Agents CLI Evaluation"
description: "The Agents CLI eval namespace supports a hill-climbing quality loop through iterative generate, grade, compare, analyze, and optimize commands. Starting with a four-case dataset expanded from a single default, the first eval run produces a baseline mean score of 4.0 on the custom_response_quality metric with a standard deviation of 1.41. A regression introduced by a swapped chef persona system prompt dropped the mean to 2.75, and the per-case diff from eval compare revealed tool misuse on weather questions that unit tests could not catch. The workflow supports automated prompt optimization via eval optimize using GEPA, but the candidate prompts require human review to avoid overfitting to the metric. Effective hill-climbing requires a small dataset that runs on every change, snapshotting full result files for later regression analysis, and treating summary metrics with suspicion as shown by the stdev tightening from 1.41 to 0.96 during the regression."
---

# Hill-Climbing Agent Quality with Agents CLI Evaluation

[Google](https://yomu.fyi/company/google) · Pier Paolo Ippolito, Elia Secchi · Oct 28, 2026

**Type:** Tutorial

## Summary

The Agents CLI eval namespace supports a hill-climbing quality loop through iterative generate, grade, compare, analyze, and optimize commands. Starting with a four-case dataset expanded from a single default, the first eval run produces a baseline mean score of 4.0 on the custom\_response\_quality metric with a standard deviation of 1.41. A regression introduced by a swapped chef persona system prompt dropped the mean to 2.75, and the per-case diff from eval compare revealed tool misuse on weather questions that unit tests could not catch. The workflow supports automated prompt optimization via eval optimize using GEPA, but the candidate prompts require human review to avoid overfitting to the metric. Effective hill-climbing requires a small dataset that runs on every change, snapshotting full result files for later regression analysis, and treating summary metrics with suspicion as shown by the stdev tightening from 1.41 to 0.96 during the regression.

## Context

Production agent quality requires continuous evaluation loops rather than one-shot demos, as changes to models, prompts, and tools can cause regressions that unit tests cannot catch.

## Approach / What changed

Use the Agents CLI eval namespace: create a small dataset of representative cases, run eval generate and eval grade for a baseline, then iterate through prompt or model changes, re-running the eval and using eval compare for per-case diffs. Use eval optimize for automated prompt hill-climbing with GEPA, with human review of candidate prompts. Schedule automated runs and snapshot full result files for traceability.

## Takeaways

- A small, representative dataset (e.g., four cases that run in under a minute) encourages running evals on every change, catching regressions early, whereas a large comprehensive dataset becomes a release gate or shelfware.
- Snapshotting full eval result JSON files, not just mean scores, enables detailed regression analysis with eval compare weeks later when a change surfaces, providing the per-case breakdown needed for diagnosis.
- More evals do not automatically lead to more fixes; a controlled study found coding agents using the CLI ran four times as many evals but changed the agent after an eval only as often as the baseline.

## FAQ

### How does eval compare help in regression detection?

eval compare reads two result JSON files and produces a structured diff showing which per-case scores changed and by how much, enabling precise identification of regressions that mean scores alone would miss.

### Why should the dataset be kept small for iterative evaluation?

A small dataset (e.g., four cases running in under a minute) encourages running evals on every change, whereas a large dataset becomes a release gate or shelfware that few run on each change.

### What are the limitations of the eval optimize command?

eval optimize only adjusts the agent's prompt, not tools or model, and may overfit to the metric's wording, so the candidate prompts require human review before promotion.

**Tags:** [Agents CLI](https://yomu.fyi/topic/agents-cli), [AI Agents](https://yomu.fyi/topic/ai-agents), [Evaluation](https://yomu.fyi/topic/evaluation), [Testing](https://yomu.fyi/topic/testing)

- Source: [Google](https://developers.googleblog.com/hill-climbing-agent-quality-with-agents-cli-evaluation)
- Source URL: https://developers.googleblog.com/hill-climbing-agent-quality-with-agents-cli-evaluation
- Ingested by Yomu: 2026-10-28T18:01:13.860Z

[Read original post](https://developers.googleblog.com/hill-climbing-agent-quality-with-agents-cli-evaluation)
