---
title: "Doing the right thing when things go wrong"
description: "Fin’s incident-management process is designed to reduce customer impact quickly when deployments, alarms, or customer reports indicate a disruption. It combines customer and product signals, including heartbeat metrics for workflows such as Fin’s response generation, with engineer monitoring of rollouts; a rollback can land in a little under two minutes. Responses run through incident.io and Slack, with defined engineering lead, incident commander, business lead, and resolution owner roles, plus the out-of-hours STRIKE responder group. The lifecycle covers triage, investigation, mitigation, and learning, with a stated preference for rolling back to a known state before fixing forward and with explicit alternatives when rollback is unavailable. After recovery, bounded follow-ups and incident reviews aim to improve recurrence prevention, blast-radius reduction, detection, and response, while early escalation and clear communication make the process repeatable rather than dependent on heroics."
---

# Doing the right thing when things go wrong

[Intercom](https://yomu.fyi/company/intercom) · Mark Gorman · Jul 14, 2026

**Type:** Explainer

## Summary

Fin’s incident-management process is designed to reduce customer impact quickly when deployments, alarms, or customer reports indicate a disruption. It combines customer and product signals, including heartbeat metrics for workflows such as Fin’s response generation, with engineer monitoring of rollouts; a rollback can land in a little under two minutes. Responses run through incident.io and Slack, with defined engineering lead, incident commander, business lead, and resolution owner roles, plus the out-of-hours STRIKE responder group. The lifecycle covers triage, investigation, mitigation, and learning, with a stated preference for rolling back to a known state before fixing forward and with explicit alternatives when rollback is unavailable. After recovery, bounded follow-ups and incident reviews aim to improve recurrence prevention, blast-radius reduction, detection, and response, while early escalation and clear communication make the process repeatable rather than dependent on heroics.

## Context

Customer-impacting incidents can interrupt conversations and create uncertainty for customers relying on Fin and Intercom. The process addresses the risk that incidents will spiral, remain unactioned, or depend on individual heroics without shared roles, tools, and response habits.

## Approach / What changed

Fin uses customer reports, workflow heartbeat metrics, alarms, and engineer monitoring to detect incidents, then coordinates response through incident.io and Slack. Defined roles, STRIKE out-of-hours coverage, a triage-investigate-mitigate-learn lifecycle, rapid rollbacks, communication, and bounded follow-ups structure the response.

## Takeaways

- Heartbeat metrics monitor whether users can complete core workflows, such as generating responses, rather than only whether servers are running.
- Fin’s responders generally prefer rolling back to a known state before fixing forward; when rollback is unavailable, they evaluate alternatives such as feature-flag changes, queue draining, or dependency bypasses.
- Post-incident follow-ups are expected to be specific and bounded, improving recurrence prevention, blast-radius reduction, detection, or response and investigation.

**Tags:** [Incident Response](https://yomu.fyi/topic/incident-response), [Monitoring](https://yomu.fyi/topic/monitoring), [Reliability](https://yomu.fyi/topic/reliability)

- Source: [Intercom](https://www.intercom.com/blog/doing-the-right-thing-when-things-go-wrong)
- Source URL: https://www.intercom.com/blog/doing-the-right-thing-when-things-go-wrong
- Ingested by Yomu: 2026-09-09T08:19:54.972Z

[Read original post](https://www.intercom.com/blog/doing-the-right-thing-when-things-go-wrong)