Building a better Bugbot

Summary

Bugbot is a code review agent built to identify logic bugs, performance issues, and security vulnerabilities in pull requests before production, as increasingly capable coding agents increased review workload. Its development moved from qualitative assessments to systematic experimentation, using an AI-driven resolution-rate metric, the BugBench benchmark, and online evaluations of real pull requests. An early production flow ran eight parallel passes with randomized diff ordering, majority voting, category filters, validation, and deduplication, while later versions adopted a fully agentic design with tools and dynamic context. Across 40 major experiments, resolution rate rose from 52% to over 70%, bugs flagged per run from 0.4 to 0.7, and resolved bugs per pull request from roughly 0.2 to about 0.5. Newer versions caught more bugs without a comparable rise in false positives, and planned capabilities include Autofix, self-verification, deep research, and continuous scanning.

Takeaways

1. 1

   Eight parallel passes received different diff orderings, and majority voting treated issues found independently by several passes as stronger signals.

2. 2

   The resolution-rate metric uses AI at PR merge time to determine which reported bugs were resolved in the final code; internal spot-checking found it classified nearly all examples correctly.

3. 3

   Switching to a fully agentic architecture produced the largest gains, enabling tool calls, dynamic context retrieval, and more aggressive prompts that investigated suspicious patterns.