Grab

Southeast Asia's leading everyday "super-app" that provides ride-hailing, food delivery, grocery shopping, and digital financial services.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

From their blog

Latest articles

Grab
Grab You Some Post-Mortem Reports
Grab describes how its service-oriented architecture makes cross-team production debugging difficult because engineers may lack familiarity with other teams’ code and architecture. Post-mortem reports are presented as the glue for shared understanding, but a sample incident entry lacks service context, bug detail, impact, chronology, and investigation steps. The proposed standard has four requirements—Chronology, Context, Empowerment, and Solutions—covering timelines, service internals, blameless educational diagnosis, and remedies across People, Product, and Process. A template includes initial symptoms, a timestamped timeline, log-based investigation with code evidence, temporary and permanent fixes, and improvement actions, while peer review by another team is required before finalization.

Grab
The Curious Case of the Phantom Instance
Datadog dashboards for the grab_attention cluster displayed periodic 1.5X step increases in Elastic Load Balancer (ELB) health check requests and ElastiCache Redis connections, creating the illusion of an untracked instance running outside Auto Scaling Group records. Inspecting instance hostname tags revealed that the existing two instances were simply receiving elevated ping counts from the load balancers. AWS Support clarified that ELB scaling events provision new nodes while keeping old nodes running for roughly 90 minutes to handle cached DNS clients. Investigation also revealed that two separate ELBs were attached to the cluster, altering expected request baselines. Furthermore, the application's health check endpoint initiated a non-pooled Redis connection on every request, directly translating load balancer pings into database connection spikes.

The best engineering reads, once a week.

Southeast Asia's leading everyday "super-app" that provides ride-hailing, food delivery, grocery shopping, and digital financial services.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

From their blog

Grab
Grab You Some Post-Mortem Reports
Grab describes how its service-oriented architecture makes cross-team production debugging difficult because engineers may lack familiarity with other teams’ code and architecture. Post-mortem reports are presented as the glue for shared understanding, but a sample incident entry lacks service context, bug detail, impact, chronology, and investigation steps. The proposed standard has four requirements—Chronology, Context, Empowerment, and Solutions—covering timelines, service internals, blameless educational diagnosis, and remedies across People, Product, and Process. A template includes initial symptoms, a timestamped timeline, log-based investigation with code evidence, temporary and permanent fixes, and improvement actions, while peer review by another team is required before finalization.

Grab
The Curious Case of the Phantom Instance
Datadog dashboards for the grab_attention cluster displayed periodic 1.5X step increases in Elastic Load Balancer (ELB) health check requests and ElastiCache Redis connections, creating the illusion of an untracked instance running outside Auto Scaling Group records. Inspecting instance hostname tags revealed that the existing two instances were simply receiving elevated ping counts from the load balancers. AWS Support clarified that ELB scaling events provision new nodes while keeping old nodes running for roughly 90 minutes to handle cached DNS clients. Investigation also revealed that two separate ELBs were attached to the cluster, altering expected request baselines. Furthermore, the application's health check endpoint initiated a non-pooled Redis connection on every request, directly translating load balancer pings into database connection spikes.

The best engineering reads, once a week.