Loading post…
SentryDominik Buszowiecki, Shaun KaastenExplainer
Debugging our AI search assistant with agent tracing
Summary
Sentry’s Search Query Assistant converts natural-language prompts into Sentry Search Syntax, but its non-deterministic output requires evals and debugging when generated queries are wrong. A production issue caused searches on custom numerical attributes to return no events, so the team used Sentry’s AI Conversation view and Agents filters to find the relevant trace, inspect the timeline, and follow tool calls into the backend API. The trace showed that tool results were being incorporated into the system prompt with the field name bug__predictions_count instead of the required tags[bug_predictions_count, number] form. The team added an eval scenario, fixed the prompt generation, verified the corrected query locally and in production, and confirmed the updated prompt and API parameter in the trace. They also instrumented local eval runs to send data to Sentry, allowing debugging before a pull request.
Context
The Search Query Assistant helps users avoid writing Sentry Search Syntax by converting natural-language prompts into queries. Because its output is non-deterministic, the team uses evals to measure expected query generation and investigates failures through local results or Sentry’s AI Conversation view when behavior depends on tool data, model versions, or infrastructure.
Approach / What changed
The team filtered the Agents view by time range, agent, and email to locate an interaction, then inspected its timeline, system prompt, LLM generation, and tool calls. They followed the related backend trace to identify incorrect prompt construction, added an eval scenario, corrected the field representation, verified the fix locally and in production, and sent local eval traces to Sentry for earlier investigation.
Takeaways
The failing query used bug__predictions_count in the system prompt, but Sentry Search Syntax required tags[bug_predictions_count, number] for the custom numerical attribute.
AI Conversation timelines expose the system prompt, LLM generation, and tool calls, while linked traces show the backend API request and the data returned to the agent.
Sending local eval runs to Sentry lets the team inspect agent behavior before opening a pull request, using the same tracing workflow as production debugging.
FAQ
How did the team find the cause of the empty search results?
They located the interaction in the Agents view, inspected its timeline, and followed the related trace to the backend API call. This showed that the system prompt listed the custom field as bug__predictions_count instead of using the required tags[bug_predictions_count, number] syntax.
Why were evals needed for the Search Query Assistant?
The assistant’s output is non-deterministic, so evals measure whether it produces the expected query for core scenarios. The team added a scenario for this bug and used it to reproduce and verify the fix.
What can be inspected in an AI Conversation trace?
The view shows the system prompt, LLM generation, and tool calls involved in producing the response. A linked trace also shows the backend API call made by the agent and the data returned by that call.
Source and freshness
- Source
- blog.sentry.io
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.