---
title: "How to Make Chip-Design Agents Earn Their Outputs"
description: "A five-checkpoint loop for chip-design teams: pin reference artifacts, trace multistage agent runs, refine tool descriptions, and re-run a composite benchmark so outputs are verified, not merely plausible."
---

# How to Make Chip-Design Agents Earn Their Outputs

**Type:** [How to](https://yomu.fyi/visual-explainers/category/how-to.md)

![A left-to-right five-step verification loop for chip-design agents: reference artifacts and checks, multistage agent run, trajectory review, tool-description refinement, and composite benchmark re-run with score, turns, and token metrics.](https://yomufyi.s3.us-east-1.amazonaws.com/visual-explainers/9f704c76-7559-4fe5-adf1-28d09d564f9f/image-6f10e54d-03a3-4bef-8993-c28b8d92cc51.webp)

> A five-checkpoint loop for chip-design teams: pin reference artifacts, trace multistage agent runs, refine tool descriptions, and re-run a composite benchmark so outputs are verified, not merely plausible.

## Explanation

Chip-design agents can produce plausible but incorrect EDA artifacts. This explainer lays out a practical verification loop: define checks against reference outputs, run the agent in a multistage environment, inspect trajectories for failure modes, tighten skills and MCP tool descriptions, then re-run a composite benchmark and compare score, turns, and tokens. The goal is not more agent activity but fewer, more reliable turns that produce artifacts engineers can sign off on.

## FAQ

### What should a chip-design agent verifier check?

Start with the final artifact against a known reference. In the Siemens Fuse EDA example, the NeMo Gym verifier checks coverage, validity, relative validation, integrity, and accuracy of generated Liberty files; track the composite score, not just whether the agent finished.

### Why refine tool descriptions instead of only prompting the model?

The NeMo Gym case used trajectory analysis to find failure modes, then refined the agent’s skills and MCP tool descriptions. After that, the composite benchmark rose from about 55% to 95%, while turns fell from 155 to 53 and output tokens from roughly 81K to 29K, showing that clearer tools can reduce wasted work.

### How do traces help debug an unreliable chip-design agent?

Traces expose model requests and tool calls as spans so you can see missing context or failed tool calls rather than guessing from the final output. Compare baseline and revised runs on the same traces to measure whether a change improves reliability or just shifts the failure.

**Tags:** [Agents](https://yomu.fyi/topic/agents.md), [Benchmark](https://yomu.fyi/topic/benchmark.md), [Chip Design](https://yomu.fyi/topic/chip-design.md), [Tool Calling](https://yomu.fyi/topic/tool-calling.md), [Tracing](https://yomu.fyi/topic/tracing.md), [Verification](https://yomu.fyi/topic/verification.md)

## Related reading

### [Claude Platform now supports native OpenTelemetry observability in Databricks MLflow](https://yomu.fyi/post/claude-platform-now-supports-native-opentelemetry-observability-in-dat.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Yuki Watanabe
- Published: Nov 3, 2026
- Type: Announcement

The Claude Platform now supports native OpenTelemetry export to Databricks MLflow, enabling developers to trace agent executions. Traces record model requests and tool calls as spans, ingested via Zerobus and stored in Unity Catalog tables linked to MLflow experiments. Setting up requires creating an experiment with Unity Catalog trace storage, configuring native export with a table-scoped OAuth token, and running a task to inspect traces. Developers can debug unexpected answers by analyzing traces with Genie Code, which identifies missing context or failed tool calls, and MLflow Issues detects recurring problems across traces. Agent Evaluation compares baseline and revised agents using the same traces to measure improvements or regressions, with domain experts able to review traces via Review Queues.


### [Finding performance wins in our own traces with a coding agent](https://yomu.fyi/post/finding-performance-wins-in-our-own-traces-with-a-coding-agent.md)
- Company: [Sentry](https://yomu.fyi/company/sentry.md)
- Author: Scott Cooper
- Published: Nov 3, 2026
- Type: Problem & solution

An engineer used a coding agent connected to Sentry MCP to inspect Sentry’s own traces and find backend performance problems. The agent sorted spans by execution frequency and duration, then returned candidate fixes with a file and line, estimated weekly volume, and an example trace. Django query spans included SQL as their description, while query-source attributes supplied file and line and the user\_agent attribute exposed a sentry-cli set-commits workload paging through roughly 16k repositories every 40 seconds. For SQL changes, the agent ran EXPLAIN ANALYZE before and after on a read-only database mirror and compared results. Fixes included reusing a loaded project to remove an estimated 143M weekly SELECTs, rewriting a GroupHistory lookup to probe each status separately, and eager-loading Sentry app relations; the GroupHistory query fell from 935ms to 0.4ms with a warm cache.


### [CWE-bench v2: More coverage, quality, and transparency](https://yomu.fyi/post/cwe-bench-v2-more-coverage-quality-and-transparency.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Gonzalo Gonzalez, Ajay Jagannath, Nazneen Rajani, Nathan Habib
- Published: Nov 3, 2026
- Type: Announcement

CWE-bench v2 is a public benchmark of 100 tasks that tests AI agents on finding and fixing security weaknesses in real open-source repositories, with fully programmatic and deterministic scoring. The tasks cover 91 different CWEs, requiring models to understand multiple vulnerability surfaces rather than recognize familiar patterns, and each task includes a repository checkout, instruction, and verifier curated by LLM judge panels and hand-reviewed. Among 11 evaluated models, Gemini 4 Argon leads with 67.8% pass@1, but even the best model fails 19% of tasks, and 95 tasks are solved by at least one model. On cost, DeepSeek V4.1 Flash achieves 57.0% pass@1 at $0.04 per rollout, Claude Opus 5.5 adds ten points at $1.83, and Gemini 4 Argon edges ahead at $3.92, forming the Pareto frontier. The benchmark is open-source and can be run via Harbor, with tasks also usable as reinforcement learning environments for training defensive agents.


### [Transformers Model Integration Contracts](https://yomu.fyi/post/transformers-model-integration-contracts.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Tarek Ziadé
- Published: Nov 3, 2026
- Type: Problem & solution

Transformers enabled PyTorch scaled dot-product attention for T5 on July 30, 2026, letting UMT5’s decoder attend to future tokens during training. The bug stayed in main for almost two months, showing why full integration tests cannot run on every pull request across more than 750 model types. The proposed Model Integration Contract combines a shrunk fixture from a real checkpoint’s configuration, a runnable usage path, and human-reviewed baselines, and holds when that path still produces the baselines. Fixtures preserve tokenizer and processor files and the original weights format, keep different layer kinds, and shrink a 1,227 MB UMT5 checkpoint to 66 MB or Llama 4 Scout from 108.6 billion parameters to 44.7 million. A five-month replay showed the UMT5 fixture flagged the regression under SDPA, while eager attention missed it; the MIC CI job reports failures without blocking pull requests.


### [Optimizing Agent Reliability and Tool Calling Efficiency for Chip Design Using NVIDIA NeMo Gym](https://yomu.fyi/post/optimizing-agent-reliability-and-tool-calling-efficiency-for-chip-desi.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Tanya Lenz
- Published: Oct 14, 2026
- Type: Problem & solution

AI agents in chip design often produce plausible but incorrect artifacts, making reliability a key barrier to production adoption. NVIDIA NeMo Gym, an open library for evaluating models and agents using environments, solves this by diagnosing failure modes in multistage workflows through verification of final outputs against reference artifacts. Applied to the Siemens Fuse EDA AI Agent for standard-cell characterization, NeMo Gym uses a verifier that checks coverage, validity, relative validation, integrity, and accuracy of generated Liberty files against known references. After refining the agent’s skills and MCP tool descriptions based on trajectory analysis of failures, the composite benchmark score improved from approximately 55% to 95%, while agent turns fell from 155 to 53 and output tokens dropped from ~81K to 29K. This method generalizes to other engineering domains by emphasizing evaluation of final artifacts and domain-specific verifiers.
