# Reading the agent traces is how you make the call your eval can't

[Sentry](https://yomu.fyi/company/sentry) · Sergiy Dybskiy · Jul 1, 2026

**Type:** Explainer

## Summary

Automated evaluations often miss unknown failure modes in AI agents because developers only write tests for anticipated errors. While building an itinerary agent for AI Engineer World's Fair, an open-weight model routed a speaker query to a tool returning only tracks, leading the model to hallucinate famous speakers and falsely claim the tool provided that data. Because a tool was executed, the trace appeared grounded until inspectable tool outputs revealed the discrepancy. To resolve this, the author retained the inexpensive model, corrected query routing, tightened system prompts against ungrounded entities, and added a custom entity-diffing eval in CI. Ultimately, directly reading agent telemetry traces alongside errors and logs is essential for diagnosing root causes and deciding trade-offs between model costs, prompt adjustments, and routing fixes.

## Context

Building an AI itinerary builder for AI Engineer World's Fair using cheap open-weight models for anonymous visitors and larger models for signed-up users, with automated evals that initially failed to anticipate specific agent hallucinations.

## Approach / What changed

Inspect agent telemetry traces directly in Sentry, fix query routing so speaker questions reach proper tools, tighten system prompts to forbid naming unreturned entities, and implement a CI groundedness check using a custom entity-matching scorer.

## Takeaways

- Tool execution can disguise hallucinations in traces unless developers inspect the actual payload returned by the tool call.
- A custom CI evaluation scorer comparing named entities against a database can verify groundedness without requiring an LLM judge.
- Automated evaluations quantify failure rates but cannot decide trade-offs among model upgrades, prompt tightening, routing repairs, or accepting errors.

**Tags:** [Developer Experience](https://yomu.fyi/topic/developer-experience), [LLMs](https://yomu.fyi/topic/llm), [Monitoring](https://yomu.fyi/topic/monitoring), [Observability](https://yomu.fyi/topic/observability), [Testing](https://yomu.fyi/topic/testing)

[Read original post](https://blog.sentry.io/spot-checking-ai-agents)