Your agent should understand what you see

Summary

Sentry's Seer Agent initially parsed user screens using DOM-scraped character-by-character ASCII grids, leading to severe token consumption, model context rot, and an inability to support interactive actions. To resolve these bottlenecks, Sentry replaced visual text scraping with a composable semantic context system where React UI components directly declare their state, metadata, and interactive affordances. A root React context provider maintains a registry of component nodes organized hierarchically through higher-order wrappers, serializing the resulting structure to JSON and backend-generated Markdown within the system prompt. Across twenty-four live pages, this approach reduced dashboard context tokens from roughly 5,500 to 1,300 while maintaining identical satisfaction rates and tool-call volumes across more than 14,000 evaluated conversations.

Takeaways

1. 1

   Switching from ASCII snapshots to semantic context dropped page context prompt usage from 85–93% down to 50–80%, reducing dashboard tokens from ~5,500 to ~1,300 and issue detail tokens from ~2,100 to ~300.

2. 2

   Across over 14,000 conversation runs evaluated by an LLM as a judge, the structured context approach matched the prior ASCII snapshot approach in both user satisfaction rates and average tool calls per conversation.

3. 3

   The composable architecture derives component hierarchy automatically from React's component tree, allowing unmigrated UI components to fall back seamlessly to ASCII snapshots.