---
title: "The Anatomy of Harness Engineering: How to Evaluate, Iterate, and Guard AI Coding Agents"
description: "The post argues that end-to-end benchmarks such as Terminal-Bench and DeepSWE show overall agent performance but rarely explain why behavior changes. It presents behavioral evaluations as fast, deterministic, unit-style checks of observable intermediate actions, such as asking clarifying questions, running validators, or invoking web search instead of guessing. The proposed workflow starts with a recent failure, uses strict milestone assertions for simple tasks and flexible outcome-based or LLM-as-a-judge checks for complex tasks, then runs batch evaluations to track aggregate pass rates despite model nondeterminism. These evaluations are positioned for the second phase of development, after an agent can dogfood its codebase, and as complementary to macro benchmarks: behavioral tests enable prompt, tool-schema, and model iteration while guarding against regressions."
---

# The Anatomy of Harness Engineering: How to Evaluate, Iterate, and Guard AI Coding Agents

[Google](https://yomu.fyi/company/google) · Taylor Mullen, Christian Gunderman · Sep 9, 2026

**Type:** Explainer

## Summary

The post argues that end-to-end benchmarks such as Terminal-Bench and DeepSWE show overall agent performance but rarely explain why behavior changes. It presents behavioral evaluations as fast, deterministic, unit-style checks of observable intermediate actions, such as asking clarifying questions, running validators, or invoking web search instead of guessing. The proposed workflow starts with a recent failure, uses strict milestone assertions for simple tasks and flexible outcome-based or LLM-as-a-judge checks for complex tasks, then runs batch evaluations to track aggregate pass rates despite model nondeterminism. These evaluations are positioned for the second phase of development, after an agent can dogfood its codebase, and as complementary to macro benchmarks: behavioral tests enable prompt, tool-schema, and model iteration while guarding against regressions.

## Context

End-to-end benchmarks provide overall performance scores but do not typically explain failures or whether prompt, tool-schema, or model changes introduced regressions. Behavioral evaluations are introduced for the second phase of agent development, after the system can handle routine developer tasks and dogfood its own codebase.

## Approach / What changed

Use a local suite of fast, deterministic behavioral checks that assert on intermediate tool calls or file modifications. Start with one observed failure, choose strict milestone assertions for simple tasks, use flexible outcome-based or LLM-as-a-judge checks for complex tasks, and automate batch runs to monitor aggregate pass rates.

## Takeaways

- Behavioral evaluations can check discrete actions such as asking clarifying questions, running a local validator, providing canonical repository links, or using web search for live information.
- Strict single-turn assertions fit simple tasks with one optimal path, while complex tasks require less rigid, outcome-based checks because correct solutions may use unexpected tool sequences.
- Batch evaluation provides a directional aggregate pass-rate signal that accommodates model nondeterminism and helps guard against regressions during prompt and model changes.

**Tags:** [AI Agents](https://yomu.fyi/topic/ai-agents), [CI/CD](https://yomu.fyi/topic/ci-cd), [Model Evaluation](https://yomu.fyi/topic/model-evaluation), [Testing](https://yomu.fyi/topic/testing)

- Source: [Google](https://developers.googleblog.com/the-anatomy-of-harness-engineering-how-to-evaluate-iterate-and-guard-ai-coding-agents)
- Source URL: https://developers.googleblog.com/the-anatomy-of-harness-engineering-how-to-evaluate-iterate-and-guard-ai-coding-agents
- Ingested by Yomu: 2026-09-09T20:01:32.017Z

[Read original post](https://developers.googleblog.com/the-anatomy-of-harness-engineering-how-to-evaluate-iterate-and-guard-ai-coding-agents)