---
title: "Debugging our AI search assistant with agent tracing"
description: "Sentry’s Search Query Assistant converts natural-language prompts into Sentry Search Syntax, but its non-deterministic output requires evals and debugging when generated queries are wrong. A production issue caused searches on custom numerical attributes to return no events, so the team used Sentry’s AI Conversation view and Agents filters to find the relevant trace, inspect the timeline, and follow tool calls into the backend API. The trace showed that tool results were being incorporated into the system prompt with the field name bug__predictions_count instead of the required tags[bug_predictions_count, number] form. The team added an eval scenario, fixed the prompt generation, verified the corrected query locally and in production, and confirmed the updated prompt and API parameter in the trace. They also instrumented local eval runs to send data to Sentry, allowing debugging before a pull request."
---

# Debugging our AI search assistant with agent tracing

[Sentry](https://yomu.fyi/company/sentry) · Dominik Buszowiecki, Shaun Kaasten · Sep 11, 2026

**Type:** Explainer

## Summary

Sentry’s Search Query Assistant converts natural-language prompts into Sentry Search Syntax, but its non-deterministic output requires evals and debugging when generated queries are wrong. A production issue caused searches on custom numerical attributes to return no events, so the team used Sentry’s AI Conversation view and Agents filters to find the relevant trace, inspect the timeline, and follow tool calls into the backend API. The trace showed that tool results were being incorporated into the system prompt with the field name bug\_\_predictions\_count instead of the required tags\[bug\_predictions\_count, number\] form. The team added an eval scenario, fixed the prompt generation, verified the corrected query locally and in production, and confirmed the updated prompt and API parameter in the trace. They also instrumented local eval runs to send data to Sentry, allowing debugging before a pull request.

## Context

The Search Query Assistant helps users avoid writing Sentry Search Syntax by converting natural-language prompts into queries. Because its output is non-deterministic, the team uses evals to measure expected query generation and investigates failures through local results or Sentry’s AI Conversation view when behavior depends on tool data, model versions, or infrastructure.

## Approach / What changed

The team filtered the Agents view by time range, agent, and email to locate an interaction, then inspected its timeline, system prompt, LLM generation, and tool calls. They followed the related backend trace to identify incorrect prompt construction, added an eval scenario, corrected the field representation, verified the fix locally and in production, and sent local eval traces to Sentry for earlier investigation.

## Takeaways

- The failing query used bug\_\_predictions\_count in the system prompt, but Sentry Search Syntax required tags\[bug\_predictions\_count, number\] for the custom numerical attribute.
- AI Conversation timelines expose the system prompt, LLM generation, and tool calls, while linked traces show the backend API request and the data returned to the agent.
- Sending local eval runs to Sentry lets the team inspect agent behavior before opening a pull request, using the same tracing workflow as production debugging.

## FAQ

### How did the team find the cause of the empty search results?

They located the interaction in the Agents view, inspected its timeline, and followed the related trace to the backend API call. This showed that the system prompt listed the custom field as bug\_\_predictions\_count instead of using the required tags\[bug\_predictions\_count, number\] syntax.

### Why were evals needed for the Search Query Assistant?

The assistant’s output is non-deterministic, so evals measure whether it produces the expected query for core scenarios. The team added a scenario for this bug and used it to reproduce and verify the fix.

### What can be inspected in an AI Conversation trace?

The view shows the system prompt, LLM generation, and tool calls involved in producing the response. A linked trace also shows the backend API call made by the agent and the data returned by that call.

**Tags:** [AI Agents](https://yomu.fyi/topic/ai-agents), [Search](https://yomu.fyi/topic/search), [Sentry](https://yomu.fyi/topic/sentry)

- Source: [Sentry](https://blog.sentry.io/debugging-our-ai-search-assistant-with-agent-tracing)
- Source URL: https://blog.sentry.io/debugging-our-ai-search-assistant-with-agent-tracing
- Ingested by Yomu: 2026-09-14T20:01:41.354Z

[Read original post](https://blog.sentry.io/debugging-our-ai-search-assistant-with-agent-tracing)