IntercomIncident / postmortem
Intercom’s outage began with a page about elevated exceptions, leading engineers to dive into Datadog and uncover a lurking 32-bit integer limit in one of the most critical parts of the app’s data model. The chaotic five-hour incident marathon required monkey patches, data migrations, and feature flag gymnastics to restore normal operations. After stabilizing the system, the team applied important technical changes to the Rails application to prevent integer overflow and similar failures. Miles McGuire presented the postmortem at Rails World 2024, sharing the debugging process and the key lessons learned from the unusual incident. The talk covers monitoring data model limits, using feature flags for safe rollouts, and enhancing application resilience to avoid future recurrence.
A 32-bit integer limit in a critical data model was the root cause of a major Intercom outage.
The incident response required a combination of monkey patches, data migrations, and feature flag toggling to restore service.
Post-incident changes aimed at making the Rails application more resilient included monitoring integer limits and enhancing data model robustness.
The outage was triggered by a 32-bit integer limit in one of the most critical parts of the app's data model, which led to elevated exceptions.
The response involved a five-hour incident marathon using monkey patches, data migrations, and feature flag gymnastics to restore normal operations.
Key lessons included the importance of monitoring data model limits, using feature flags for safe rollouts, and building application resilience to avoid similar failures.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
A/B testing trades traffic, time, and exposure risk for evidence. This explainer maps the benefit of a controlled comparison, the real cost of running one, and the condition that decides when measuring beats shipping on judgment.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
