Reading the agent traces is how you make the call your eval can't

Summary

Automated evaluations often miss unknown failure modes in AI agents because developers only write tests for anticipated errors. While building an itinerary agent for AI Engineer World's Fair, an open-weight model routed a speaker query to a tool returning only tracks, leading the model to hallucinate famous speakers and falsely claim the tool provided that data. Because a tool was executed, the trace appeared grounded until inspectable tool outputs revealed the discrepancy. To resolve this, the author retained the inexpensive model, corrected query routing, tightened system prompts against ungrounded entities, and added a custom entity-diffing eval in CI. Ultimately, directly reading agent telemetry traces alongside errors and logs is essential for diagnosing root causes and deciding trade-offs between model costs, prompt adjustments, and routing fixes.

Context

Building an AI itinerary builder for AI Engineer World's Fair using cheap open-weight models for anonymous visitors and larger models for signed-up users, with automated evals that initially failed to anticipate specific agent hallucinations.

Approach / What changed

Inspect agent telemetry traces directly in Sentry, fix query routing so speaker questions reach proper tools, tighten system prompts to forbid naming unreturned entities, and implement a CI groundedness check using a custom entity-matching scorer.

Takeaways

• Tool execution can disguise hallucinations in traces unless developers inspect the actual payload returned by the tool call.
• A custom CI evaluation scorer comparing named entities against a database can verify groundedness without requiring an LLM judge.
• Automated evaluations quantify failure rates but cannot decide trade-offs among model upgrades, prompt tightening, routing repairs, or accepting errors.

Related reading

• Sentry · Jun 5, 2026

  Errors, traces, logs, metrics: when to reach for what

  Modern application telemetry relies on four overlapping signals: errors, traces, logs, and metrics. Each telemetry type addresses distinct engineering questions and powers specific debugging workflows. Errors track grouped, actionable application crashes, whereas traces map timing and execution paths across dependency waterfalls. Metrics aggregate numerical measurements across historical deployments, and logs preserve complete state and decision-making context at specific moments in execution. Emitting telemetry directly into the format needed for a given workflow proves superior to relying on a single wide event, as tools like AI coding agents require structured spans to optimize execution paths while human debuggers rely on unsampled logs and metrics to pinpoint silent failures.

  Sergiy Dybskiy

• Sentry · Jul 9, 2026

  When and what should I be logging?

  Targeted structured logging provides a fast way to capture operational behavior in production without requiring new deployments for debugging. Developers benefit most from logging key runtime decisions, multi-step algorithm outcomes, mutating audit operations, and context around non-critical or retried errors. Rather than outputting plain text, applications should record structured key-value pairs that capture who performed an action, what occurred, and associated trace identifiers. Sensitive information such as secrets and personally identifiable data must be excluded or scrubbed, and large unparsed payloads should be avoided to prevent excessive costs. Managing log levels appropriately and treating logs as potentially temporary instrumentation helps maintain actionable telemetry without generating unnecessary noise.