Works on my machine: how we use AI to reproduce reported bugs

Summary

Sentry maintains 159 open-source SDK packages across multiple languages, leading to high triage volume and significant boilerplate setup overhead when reproducing reported bugs. To streamline this workflow, the team developed a Claude skill that parses GitHub issue URLs to extract language, framework, and SDK versions. The skill generates a reproduction branch, sets up minimal boilerplate using ecosystem package managers like uv or npm, attempts execution, and opens a pull request with explicit run instructions. By allowing the model to exit early when encountering overly complex environments, the tool reliably prepares reproducible playgrounds without stumbling on edge cases. This workflow saved several hours in testing scenarios like Django HTTPS proxy bugs while avoiding excessive bot noise through scoped human interaction.

Takeaways

1. 1

   Providing an explicit early exit option in LLM prompts prevents agents from stumbling through overly complex tasks and encourages clear explanations of reproduction challenges.

2. 2

   Structuring Claude skill Markdown files into discrete steps, integrating CLI tools like gh, and defining explicit error-handling boundaries increases agent reliability.

3. 3

   SDK teams avoided full automated triage execution on GitHub issues to prevent excessive bot noise and retain necessary human oversight.