Implementing ChatOps into our Incident Management Procedure

Summary

Shopify’s incident management process separates coordination from technical remediation: the incident manager on call (IMOC) leads response, while component experts fix production issues and a Support Response Manager handles public communication. The process follows a simplified Incident Command System hierarchy and moves from detection and confirmation through communication, mitigation, verification, documentation, investigation, and root cause analysis. Its ChatOps chatbot, Spy, runs in Slack through Lita and integrates with PagerDuty, StatusPage, and GitHub via webhooks, providing commands for paging, incident startup, status communication, mitigation, and stopping incidents. Spy binds incidents to a shared #war-room, records notes and event times, generates service disruption documents, sends status-page reminders, and can summon the IMOC squad when an incident lasts beyond a specified period. The result is a more consistent response that reduces context switching and post-incident toil while supporting detailed post-mortems.

Takeaways

1. 1

   The IMOC coordinates communication, escalation, follow-through, and incident progression; component experts, rather than the IMOC, are responsible for fixing production issues.

2. 2

   Spy binds each incident to a #war-room Slack channel, giving responders one place for discussions and visible operational actions such as failover, deploy locks, and traffic rebalancing.

3. 3

   Spy captures war-room notes and event times in service disruption documents, reminds IMOCs about stale status pages, and contacts the IMOC squad to help prevent on-call fatigue.

FAQ