How I hunt for vulnerabilities with AI

Summary

The author describes using AI-assisted code review to hunt for vulnerabilities in ClickHouse, a large C++ codebase, despite having limited bug-bounty experience. GitHub Copilot, Claude, ChatGPT, and Gemini models were used to inspect code, generate vulnerability hypotheses, explain unfamiliar subsystems, and identify older or newly introduced attack surfaces. Promising ideas were recorded, then investigated with Python scripts against a locally running ClickHouse Docker container, followed by manual review and reproduction to distinguish genuine findings from hallucinations and misconfigurations. The work produced a few real vulnerabilities that were reported to ClickHouse’s security team and bug bounty program, while the broader conclusion is that AI can accelerate exploration and validation but still requires persistent questioning and human verification.

Takeaways

1. 1

   AI-assisted review can expose unfamiliar code paths and generate vulnerability hypotheses, but repeated ideas, hallucinations, and misconfiguration claims must be filtered by inspecting the source and documentation.

2. 2

   The validation workflow asks Copilot to create and iteratively correct Python reproductions against a running local ClickHouse Docker container, followed by manual execution and review to check for false positives.

3. 3

   The author reported a few confirmed vulnerabilities to ClickHouse’s security team and bug bounty program, while emphasizing that the exact prompting strategy may change as language models evolve.