Grab You Some Post-Mortem Reports

Summary

Grab describes how its service-oriented architecture makes cross-team production debugging difficult because engineers may lack familiarity with other teams’ code and architecture. Post-mortem reports are presented as the glue for shared understanding, but a sample incident entry lacks service context, bug detail, impact, chronology, and investigation steps. The proposed standard has four requirements—Chronology, Context, Empowerment, and Solutions—covering timelines, service internals, blameless educational diagnosis, and remedies across People, Product, and Process. A template includes initial symptoms, a timestamped timeline, log-based investigation with code evidence, temporary and permanent fixes, and improvement actions, while peer review by another team is required before finalization.

Takeaways

1. 1

   Chronology records events, response times, downtime impact, pager bottlenecks, and possible flaws in metric alerts.

2. 2

   Empowerment requires blameless reports that teach readers how problems were diagnosed, including the relevant log queries, conclusions, and technical evidence.

3. 3

   Improvements are grouped into People, Product, and Process issues, covering communication, service robustness and alerts, and operational procedures.