---
title: "How Agent Tracing Turns One Run Into a Decision Loop"
description: "Agent tracing records prompts, tool calls, and outputs as one span-linked run, then turns that record into eval datasets, alerts, and regression checks. Here is the five-stage flow from capture to decision."
---

# How Agent Tracing Turns One Run Into a Decision Loop

**Type:** [How it works](https://yomu.fyi/visual-explainers/category/how-it-works.md)

![A left-to-right five-stage diagram: capture prompts, tool calls, and outputs as spans; assemble one trace per run; flag silent failures, slow steps, and lost context; export conversations as eval datasets; route spans to an extra destination, with a return arrow closing the loop.](https://yomufyi.s3.us-east-1.amazonaws.com/visual-explainers/68bba876-5132-419a-b2c6-de19b68e7c29/image-7fd791d9-51a9-4730-a3f8-ecfc60ddc2a8.webp)

> Agent tracing records prompts, tool calls, and outputs as one span-linked run, then turns that record into eval datasets, alerts, and regression checks. Here is the five-stage flow from capture to decision.

## Explanation

Agent tracing is not just logging: it stitches each prompt, tool call, and output into a single span-linked run so you can see where an agent went quiet, slow, or lost context. The flow starts with instrumented capture, assembles one trace per run, and then reads failures off the timeline. From there, real conversations become eval datasets, and the same spans can be routed into an existing observability pipeline as an extra destination rather than a replacement. Engineers care because the trace that debugs a stuck loop can also catch a prompt regression that unit tests miss, as long as the trace stays accurate when a conversation is interrupted mid-stream.

## FAQ

### Why keep one trace per run instead of separate logs for prompts and tool calls?

Separate logs lose ordering and causality, so you cannot tell which tool call preceded a silent failure or a slow step. Capturing prompts, tool calls, and outputs in a single trace makes those relationships visible in multi-agent workflows. It also surfaces context loss, where an agent quietly drops information it was given earlier.

### How do production traces become eval datasets without hand-labeling?

You export real production conversations and record the first user message as the input and the last agent reply as the expected output. Grading against that set gives you a repeatable baseline instead of ad hoc spot checks. Per-case diffs matter: in one Agents CLI run a swapped system prompt dropped the mean score from 4.0 to 2.75 and revealed tool misuse that unit tests never caught.

### Does adding agent tracing mean replacing your existing observability pipeline?

No. OpenTelemetry spans can be sent to an extra destination, so existing pipelines keep running unchanged while agent-specific views are added alongside them. The gotcha is trace fidelity: if an interrupted conversation is not truncated to what the caller actually heard, the exported eval data inherits that error as ground truth.

**Tags:** [Agent Tracing](https://yomu.fyi/topic/agent-tracing.md), [Evaluation](https://yomu.fyi/topic/evaluation.md), [Observability](https://yomu.fyi/topic/observability.md), [OpenTelemetry](https://yomu.fyi/topic/opentelemetry.md), [Tool Calling](https://yomu.fyi/topic/tool-calling.md), [Tracing](https://yomu.fyi/topic/tracing.md)

## Related reading

### [Voice Agents are here. Context turns them into real customer value.](https://yomu.fyi/post/voice-agents-are-here-context-turns-them-into-real-customer-value.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Sathish Gangichetty, Rohit Bhagwat, Usman Zubair
- Published: Oct 28, 2026
- Type: Explainer

Voice agents are now technically feasible because synthetic voices sound natural, speech-to-text is real-time, and agents can keep pace with human conversational timing. However, most pilots prove an agent can hold a conversation without proving it can be trusted with a real customer. The missing element is context: knowing who is calling, what is already in motion, and what the agent is allowed to do. Databricks provides this brain by combining customer context, an ontology of business relationships, and governance rules that already live on its platform. The agent uses fast decision models (System 1) for millisecond choices and large language models (System 2) for careful reasoning, with every decision recorded for traceability and improvement.


### [Five things you can do with Sentry’s Agent Tracing](https://yomu.fyi/post/five-things-you-can-do-with-sentry-s-agent-tracing.md)
- Company: [Sentry](https://yomu.fyi/company/sentry.md)
- Author: Anton Bjorkman
- Published: Oct 28, 2026
- Type: Tutorial

Sentry’s Agent Tracing captures prompts, tool calls, and outputs in a single trace, enabling debugging of silent failures, slow steps, and context loss in multi-agent workflows. The post details five recipes: exporting production conversations into eval datasets via a CLI command that records the first user message as input and the last agent reply as output. Another recipe automates error triage by having Claude’s scheduled tasks pull stats via the MCP server and file tickets in Linear for broken tools, stuck loops, or costly runs. Additional recipes cover tracking per-call costs with the Vercel AI SDK, building dashboards and alerts for spend monitoring, and sending OpenTelemetry spans to Sentry as an extra destination without replacing existing pipelines.


### [Automate your Supabase backend with Make](https://yomu.fyi/post/automate-your-supabase-backend-with-make.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: elsa\_heffernan
- Published: Oct 28, 2026
- Type: Announcement

The Supabase integration for Make is now available, connecting Supabase projects to Make’s visual workflow builder. Users can query databases, read logs, deploy Edge Functions, and manage projects as workflow steps. The integration offers two connection methods: pre-built Supabase API apps for deterministic steps, and an MCP client for AI agents that get structured access to projects. Scenarios work with live data without requiring exports, eliminating the need for nightly syncs. Make tracks every scenario run and agent tool call, and each MCP tool is annotated as read-only or destructive for governance.


### [Hill-Climbing Agent Quality with Agents CLI Evaluation](https://yomu.fyi/post/hill-climbing-agent-quality-with-agents-cli-evaluation.md)
- Company: [Google](https://yomu.fyi/company/google.md)
- Author: Pier Paolo Ippolito, Elia Secchi
- Published: Oct 28, 2026
- Type: Tutorial

The Agents CLI eval namespace supports a hill-climbing quality loop through iterative generate, grade, compare, analyze, and optimize commands. Starting with a four-case dataset expanded from a single default, the first eval run produces a baseline mean score of 4.0 on the custom\_response\_quality metric with a standard deviation of 1.41. A regression introduced by a swapped chef persona system prompt dropped the mean to 2.75, and the per-case diff from eval compare revealed tool misuse on weather questions that unit tests could not catch. The workflow supports automated prompt optimization via eval optimize using GEPA, but the candidate prompts require human review to avoid overfitting to the metric. Effective hill-climbing requires a small dataset that runs on every change, snapshotting full result files for later regression analysis, and treating summary metrics with suspicion as shown by the stdev tightening from 1.41 to 0.96 during the regression.


### [From spot fixes to measured gains: improving document extraction accuracy with ai\_extract and MLflow Prompt Registry](https://yomu.fyi/post/from-spot-fixes-to-measured-gains-improving-document-extraction-accura.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Kristen Drexinger, Seshendranath Balla
- Published: Oct 27, 2026
- Type: Problem & solution

Document extraction across dozens of PDF layouts is unreliable when fields are tuned in isolation; fixing one field can cause hidden regressions and apparent improvements may stem from normal run-to-run variation. The project extracting roughly 40 structured fields from partner-submitted order documents used human reviewers who corrected and approved extracted values, providing ground truth for evaluation. Field-specific rules go in JSON schema descriptions, document-level reasoning in a global prompt, precision mode handles complex structures, and citations diagnose failures. Candidates are versioned in MLflow Prompt Registry with aliases and evaluated against fixed ground truth using three runs to filter noise before promotion, outputting a BETTER, WORSE, or INCONCLUSIVE verdict. This approach reduced total errors by 32% and revision errors by 64%, creating a repeatable and regression-safe extraction workflow now automated by Genie Code.
