---
title: "Automated agent triage with Agent Tracing and Claude Routines"
description: "Evaluating Sentry's AI agent Seer previously required manually querying evaluation data and parsing raw JSON traces to find tool failures, latency spikes, and hallucinations. To automate this triage, the team deployed a daily Claude Routine connected to Sentry using a Model Context Protocol server. Each morning, the routine analyzes hundreds of overnight agent conversations, reviews aggregate error metrics, samples failed and successful conversation spans, and inspects the agent codebase for context. Claude evaluates whether final decisions match underlying reasoning and creates Linear tickets for newly discovered issues. This daily automated analysis accelerated the feedback loop, allowing engineers to identify efficiency issues like repeated malformed repository queries and deploy same-day fixes."
---

# Automated agent triage with Agent Tracing and Claude Routines

[Sentry](https://yomu.fyi/company/sentry) · Trevor Elkins · Aug 13, 2026

**Type:** Problem & solution

## Summary

Evaluating Sentry's AI agent Seer previously required manually querying evaluation data and parsing raw JSON traces to find tool failures, latency spikes, and hallucinations. To automate this triage, the team deployed a daily Claude Routine connected to Sentry using a Model Context Protocol server. Each morning, the routine analyzes hundreds of overnight agent conversations, reviews aggregate error metrics, samples failed and successful conversation spans, and inspects the agent codebase for context. Claude evaluates whether final decisions match underlying reasoning and creates Linear tickets for newly discovered issues. This daily automated analysis accelerated the feedback loop, allowing engineers to identify efficiency issues like repeated malformed repository queries and deploy same-day fixes.

## Context

Evaluating Seer, Sentry's AI agent for triaging and fixing errors, previously required a manual, friction-heavy workflow involving hand-rolled parsers for raw JSON trace data. The team needed a connected, automated way to detect tool call failures, hallucinated outputs, latency spikes, agent loops, and cost anomalies across conversations, while also cross-checking successful runs against their reasoning.

## Approach / What changed

Sentry set up a daily Claude Routine integrated with a Sentry Model Context Protocol (MCP) server. The routine queries Agent Tracing data for overnight conversations, pulls aggregate tool and conversation metrics, samples failure and success spans, and consults the Seer codebase. It verifies whether agent verdicts match reasoning and searches Linear to file tickets for any previously untracked issues.

## Takeaways

- Sampling both successful and failed conversations is critical because runs without tool errors can still reach incorrect verdicts.
- Combining aggregate error metrics with sampled individual trace spans helped uncover systematic patterns, such as the agent repeatedly guessing malformed repository names.
- Offloading trace-querying mechanics to the Sentry MCP server allowed the routine prompt to stay simple and focus strictly on judgment and issue triage.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Monitoring](https://yomu.fyi/topic/monitoring), [Observability](https://yomu.fyi/topic/observability), [Reliability](https://yomu.fyi/topic/reliability)

- Source: [Sentry](https://blog.sentry.io/claude-routines-agent-triage)
- Source URL: https://blog.sentry.io/claude-routines-agent-triage
- Ingested by Yomu: 2026-08-27T14:52:34.360Z

[Read original post](https://blog.sentry.io/claude-routines-agent-triage)