---
title: "Use NVIDIA NeMo to Turn Agent Failures into Tests and Harness Fixes"
description: "Generic benchmarks often fail to diagnose why an agent chooses the wrong tool, misses context, or completes a task without meeting domain requirements. The NemoClaw optimization loop uses NeMo Compass to analyze recorded traces against intended behavior defined in ETHOS.md and surface recurring problems such as the agent sending a message without user confirmation. NeMo Eval Author then creates custom Harbor evaluations that reproduce the identified failure, grounding tests in real agent behavior rather than synthetic examples. The harness fix—a change to the agent’s SOUL.md system prompt—instructs it to prepare for review and stop before sending, and evaluation results show the agent passes all development and held-out tasks while reducing tool calls. In an engineering agent example, the same loop reduced malformed tool calls from 142 to zero and achieved a 28 percent relative improvement in workflow score."
---

# Use NVIDIA NeMo to Turn Agent Failures into Tests and Harness Fixes

[NVIDIA](https://yomu.fyi/company/nvidia-developer-blog) · Patrick Moorhead · Nov 4, 2026

**Type:** Tutorial

## Summary

Generic benchmarks often fail to diagnose why an agent chooses the wrong tool, misses context, or completes a task without meeting domain requirements. The NemoClaw optimization loop uses NeMo Compass to analyze recorded traces against intended behavior defined in ETHOS.md and surface recurring problems such as the agent sending a message without user confirmation. NeMo Eval Author then creates custom Harbor evaluations that reproduce the identified failure, grounding tests in real agent behavior rather than synthetic examples. The harness fix—a change to the agent’s SOUL.md system prompt—instructs it to prepare for review and stop before sending, and evaluation results show the agent passes all development and held-out tasks while reducing tool calls. In an engineering agent example, the same loop reduced malformed tool calls from 142 to zero and achieved a 28 percent relative improvement in workflow score.

## Context

Improving an agent for a domain-specific task requires more than model tuning; prompts, tools, routing, memory, skills, and harness code all shape production reliability. Generic benchmark scores may not explain why an agent chooses the wrong tool, follows the wrong workflow branch, misses relevant context, or completes a task without meeting a domain-specific requirement. Traces help investigate failures, while evaluations help measure whether a change improves behavior.

## Approach / What changed

Use NeMo Compass to analyze recorded agent traces against intended behavior defined in ETHOS.md, surfacing recurring failures such as sending messages without user confirmation. Use NeMo Eval Author to create custom Harbor evaluations that reproduce each identified failure. Update the agent harness—for example, modify SOUL.md to instruct the agent to prepare drafts for review and stop before sending. Rerun evaluations on development and held-out tasks to measure improvement and check for regressions. Repeat the loop on new traces.

## Takeaways

- NeMo Compass analyzes agent traces against a behavior definition file (ETHOS.md) to identify recurring failures, such as sending a message without user confirmation, without requiring manual inspection of each run.
- NeMo Eval Author helps create custom evaluations that reproduce real failures found by NeMo Compass, grounding tests in actual agent behavior rather than synthetic examples.
- Hold out some evaluation tasks from the coding agent to assess whether a harness change generalizes beyond development cases; accept changes only when they improve both test and held-out sets.

## FAQ

### How does NeMo Compass identify issues in agent traces?

NeMo Compass analyzes traces against multiple signals: signs of unusual behavior, how the agent uses tools, and divergence from intended behavior defined in ETHOS.md. It outputs findings with supporting trace IDs, reducing a large trace corpus to a structured backlog of issues.

### What is the purpose of holding out evaluation tasks?

Held-out tasks are withheld from the coding agent proposing harness changes to assess whether improvements generalize beyond development cases. This helps avoid overfitting to the test set.

### Can this loop reduce tool calls in engineering agents?

Yes. In a car door design example, NeMo Compass identified malformed tool calls; a coding agent added validation, reducing malformed calls from 142 to zero and overall tool calls per run by 35 percent.

**Tags:** [AI Agents](https://yomu.fyi/topic/ai-agents), [Evaluation](https://yomu.fyi/topic/evaluation), [Nemo](https://yomu.fyi/topic/nemo)

- Source: [NVIDIA](https://developer.nvidia.com/blog/use-nvidia-nemo-to-turn-agent-failures-into-tests-and-harness-fixes)
- Source URL: https://developer.nvidia.com/blog/use-nvidia-nemo-to-turn-agent-failures-into-tests-and-harness-fixes
- Ingested by Yomu: 2026-11-04T18:01:09.615Z

[Read original post](https://developer.nvidia.com/blog/use-nvidia-nemo-to-turn-agent-failures-into-tests-and-harness-fixes)
