Bug Bounty Year in Review 2019

Summary

Shopify’s 2019 bug bounty review describes experiments and process improvements intended to increase program speed, while reporting changes in response, resolution, disclosure, and bounty metrics. Shopify-Experiments, a private program launched in mid-2019 for high-signal, high-impact hackers, tested expanded scope, full payment after triage, disclosure requirements, self-closing false positives, and collaboration with third-party developers. Using registration data, HackerOne API exports, and a Slack chatbot, the team improved program analytics and automated report-state changes, assignments, comments, bounty suggestions, and common invalid-report closures. Average first response fell to 16 hours, triage to 2 days and 13 hours, bounty payment to 7 days and 1 hour, and post-triage resolution to 20 days and 3 hours; disclosures rose to 74 bugs. For 2020, Shopify announced full bounty payment within seven days of triage, a $50,000 maximum bounty, higher payouts for several vulnerability classes, and added visibility for duplicate reports.

Context

Shopify wanted to increase bug bounty program speed, improve its analytics, understand testing activity beyond HackerOne’s standard platform data, and evaluate process changes through a private program. The review also records efforts to improve communication with hackers and learn from peer programs.

Approach / What changed

Shopify launched the private Shopify-Experiments program, used provisioned-account registration data and regular HackerOne API exports for activity analysis, and built Slack automation for report management. It tested expanded scope, full bounty payment after validation and triage, disclosure and duplicate-report changes, self-closing false positives, and collaboration with third-party developers.

Takeaways

• Shopify-Experiments invited high-signal, high-impact hackers and used controlled experiments to assess workload, bounty handling, disclosure, false-positive closure, and third-party app testing before public-program changes.
• Slack automation and HackerOne API calls supported report-state changes, assignments, comments, bounty suggestions, and quick closure of common invalid reports, contributing to faster response metrics.
• In 2019, Shopify received 1,379 reports, paid $126,100, awarded 107 bounties, and disclosed 74 bugs; the lower bounty totals were partly attributed to the exceptional H1-514 event in 2018 and the merged Shopify Scripts program.

Related reading