Reading the agent traces is how you make the call your eval can't

Summary

Automated evaluations often miss unknown failure modes in AI agents because developers only write tests for anticipated errors. While building an itinerary agent for AI Engineer World's Fair, an open-weight model routed a speaker query to a tool returning only tracks, leading the model to hallucinate famous speakers and falsely claim the tool provided that data. Because a tool was executed, the trace appeared grounded until inspectable tool outputs revealed the discrepancy. To resolve this, the author retained the inexpensive model, corrected query routing, tightened system prompts against ungrounded entities, and added a custom entity-diffing eval in CI. Ultimately, directly reading agent telemetry traces alongside errors and logs is essential for diagnosing root causes and deciding trade-offs between model costs, prompt adjustments, and routing fixes.

Takeaways

1. 1

   Tool execution can disguise hallucinations in traces unless developers inspect the actual payload returned by the tool call.

2. 2

   A custom CI evaluation scorer comparing named entities against a database can verify groundedness without requiring an LLM judge.

3. 3

   Automated evaluations quantify failure rates but cannot decide trade-offs among model upgrades, prompt tightening, routing repairs, or accepting errors.