Grab You Some Post-Mortem Reports

Summary

Grab uses a Service-Oriented Architecture to deploy features quickly, but unfamiliarity across teams makes cross-service production debugging difficult. Historical incident reports lacked context, diagnostic details, impact data, and timelines, leaving outside engineers unable to learn from past outages. To address this, Grab established a four-pillar framework for post-mortem reports covering chronology, context, empowerment, and solutions. The approach mandates blameless, educational write-ups that categorize post-incident improvements across people, product, and process dimensions. Final reports undergo peer reviews by engineers from external teams to ensure clarity and remove bias.

Context

Cross-team debugging during production issues was difficult in Grab's Service-Oriented Architecture due to unfamiliarity with other teams' code, exacerbated by brief post-mortem reports lacking context, timelines, and investigation details.

Approach / What changed

Grab established a standardized post-mortem benchmark requiring four key elements—chronology, context, empowerment, and solutions—along with peer review by engineers from outside teams.

Takeaways

• Post-mortems should provide educational, step-by-step investigation details, such as exact log queries and code context, to empower external engineers to debug future issues.
• Incident solutions and improvements must be systematically categorized into people, product, and process bottlenecks.
• Requiring engineers from another team to peer-review final post-mortems ensures adequate service context is included without personal bias.

Related reading

• Grab · Jul 31, 2019

  How We Built a Logging Stack at Grab

  Grab needed a scalable logging platform to replace slow, fragmented systems that hindered debugging across their growing service fleet. Generating 25TB of daily logs, the team built a horizontally scalable Elasticsearch cluster configured via Ansible and monitored with Datadog. Although the initial proof of concept assigned all node roles (ingest, coordinator, master, and data) to every machine, operating at scale introduced major challenges with JVM heap exhaustion and cluster stability. The team resolved memory pressure and performance bottlenecks by tuning circuit breakers, lowering field data cache limits, adjusting shard allocations based on segment memory, and disabling translog compression during shard transfers.

  Daniel Kasen

• Grab · Dec 23, 2025

  Demystifying user journeys: Revolutionizing troubleshooting with auto tracking

  Tracking user journeys across hundreds of mobile UI components via manual clickstream instrumentation frequently caused data gaps and prolonged issue resolution. Grab designed AutoTrack, an SDK that automatically records application lifecycle states, launch triggers, user interactions, and screen contexts. The system monitors transitions across foreground, background, and inactive states alongside entry points like deep links and push notifications. On the client side, AutoTrack captures complete screen hierarchy metadata from native Android activities, fragments, and iOS view controllers, redacting sensitive data to comply with privacy regulations. This automated trace enables engineering teams to reproduce elusive bugs, route incidents to domain owners, and generate automated UI test cases from real-world usage patterns.

  Alex Prokofiev

• Grab · Feb 27, 2025

  TechDocs at Grab: Cultivating a culture of quality documentation

  Engineering organizations frequently struggle with fragmented documentation, stale content, and a lack of clear ownership across disparate tools. To address these issues, Grab established TechDocs on its central Helix platform, embedding a Docs-as-Code workflow into daily engineering routines. Feedback gathered from quantitative surveys and one-on-one sessions shaped governance policies, separating stable platform documentation stored in GitLab from collaborative artifacts like RFCs in Confluence. To sustain document freshness, the platform assigns mandatory points of contact, displays last-updated timestamps, and flags pages untouched for more than three months.

  David Khu

• Grab · Sep 23, 2020

  Our Journey to Continuous Delivery at Grab (Part 1)

  Around the end of 2018, Grab's backend architecture consisted of roughly 270 services managed through fragmented, manual deployment workflows. Engineers copied release parameters between build logs, wiki pages, Slack bots, and multiple Jenkins jobs, leading to high operational friction and an average of 10 business days between production updates for a service. To streamline delivery, Grab built Conveyor, an internal automation platform built on top of open-source Spinnaker. Conveyor introduced a custom user interface focused on pipeline visibility and a pipeline-as-code DSL called Artificer using Jsonnet files in the monorepository. The platform automatically registers build artifacts with commit metadata to eliminate manual parameter entry and automatically provisions integration, staging, and production pipelines.

  Sylvain Bougerel