Loading post…
SentryTrevor ElkinsProblem & solution
Automated agent triage with Agent Tracing and Claude Routines
Summary
Evaluating Sentry's AI agent Seer previously required manually querying evaluation data and parsing raw JSON traces to find tool failures, latency spikes, and hallucinations. To automate this triage, the team deployed a daily Claude Routine connected to Sentry using a Model Context Protocol server. Each morning, the routine analyzes hundreds of overnight agent conversations, reviews aggregate error metrics, samples failed and successful conversation spans, and inspects the agent codebase for context. Claude evaluates whether final decisions match underlying reasoning and creates Linear tickets for newly discovered issues. This daily automated analysis accelerated the feedback loop, allowing engineers to identify efficiency issues like repeated malformed repository queries and deploy same-day fixes.
Context
Evaluating Seer, Sentry's AI agent for triaging and fixing errors, previously required a manual, friction-heavy workflow involving hand-rolled parsers for raw JSON trace data. The team needed a connected, automated way to detect tool call failures, hallucinated outputs, latency spikes, agent loops, and cost anomalies across conversations, while also cross-checking successful runs against their reasoning.
Approach / What changed
Sentry set up a daily Claude Routine integrated with a Sentry Model Context Protocol (MCP) server. The routine queries Agent Tracing data for overnight conversations, pulls aggregate tool and conversation metrics, samples failure and success spans, and consults the Seer codebase. It verifies whether agent verdicts match reasoning and searches Linear to file tickets for any previously untracked issues.
Takeaways
Sampling both successful and failed conversations is critical because runs without tool errors can still reach incorrect verdicts.
Combining aggregate error metrics with sampled individual trace spans helped uncover systematic patterns, such as the agent repeatedly guessing malformed repository names.
Offloading trace-querying mechanics to the Sentry MCP server allowed the routine prompt to stay simple and focus strictly on judgment and issue triage.
Source and freshness
- Source
- blog.sentry.io
- Published
- Added to Yomu
- Last updated
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.