---
title: "Reliability"
description: "289 posts about Reliability, summarised, each linking to the original."
---

# Reliability
> 289 posts about Reliability, summarised, each linking to the original.

## Articles

### [Come and #hackallthethings at Grab](https://yomu.fyi/post/come-and-hackallthethings-at-grab.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Grab Engineering
- Published: Jul 11, 2017
- Type: Announcement

Grab has officially launched a public bug bounty program in partnership with HackerOne to strengthen the security of its platform. This rollout follows a private bounty initiative operated over the previous year, during which the organization worked with over 350 security researchers and resolved nearly 200 awarded bug reports. The new public program invites external researchers to scrutinize Grab's code for critical flaws, including remote code execution, SQL injections, and exportable cross-site scripting vulnerabilities. To support ethical and responsible disclosure, Grab offers payouts reaching up to $10,000 per valid vulnerability report based on severity and impact.


### [How to Go from a Quick Idea to an Essential Feature in Four Steps](https://yomu.fyi/post/how-to-go-from-a-quick-idea-to-an-essential-feature-in-four-steps.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Da Huang
- Published: May 16, 2017
- Type: Problem & solution

Grab engineered an in-app messaging platform, GrabChat, to help drivers and passengers coordinate pickups across Southeast Asian markets characterized by weak 2G connectivity and high packet loss. The team developed an in-house TCP messaging architecture consisting of a TCP gateway named Gundam and a message dispatcher named Hermes connected to internal backend services over HTTPS. To protect backend server resources from resend loops during poor connection states, the communication protocol adopts a "server only push once" model that delegates retry handling to the client. Data science evaluations using a pre-trained cancellation prediction model confirmed that GrabChat adoption correlated with reduced booking cancellations. Following early usage feedback, the team further iterated on the feature by introducing pre-written message templates to reduce driver distraction on the road.


### [Troubleshooting Unusual AWS ELB 5XX Error](https://yomu.fyi/post/troubleshooting-unusual-aws-elb-5xx-error.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Dharmarth Shah
- Published: May 10, 2017
- Type: Problem & solution

Grab experienced intermittent HTTP 5XX alerts when its Gothena service sent driver location updates to the Astrolabe service through an AWS Elastic Load Balancer (ELB). CloudWatch metrics revealed that requests were failing to reach healthy backend instances because of an uneven load distribution favoring a single ELB node in one Availability Zone. The team verified that Route 53 was properly using Alias records and ruled out OS-level DNS caching since Linux does not cache DNS queries by default. Connection inspection with netstat across multiple Go services confirmed a heavily skewed distribution of connections toward specific ELB IP addresses. Comparative tests with cURL, tcpdump, Go, Python, and Ruby in an isolated environment demonstrated that Go reused connections across requests while other runtimes opened new connections per request.


### [A Key Expired in Redis, You Won't Believe What Happened Next](https://yomu.fyi/post/a-key-expired-in-redis-you-won-t-believe-what-happened-next.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Karan Kamath
- Published: Mar 27, 2017
- Type: Incident / postmortem

Grab experienced an issue where its Unicorn API served stale data for up to 45 to 60 minutes despite expected cache invalidation times totaling around 11 minutes. The setup utilized ElastiCache Redis 2.x configured with a single master node for writes and two read-only slaves handling reads. Investigation revealed that in Redis 2.x, slave nodes do not expire keys on their own and only delete them upon receiving an explicit DEL command from the master. Because the master only actively checks and deletes 200 random keys per second, clearing expired keys across roughly 5.6 million cached items mathematically required over 110 hours, resulting in slaves serving expired data.


### [How We Found and Fixed a Filesystem Corruption Bug](https://yomu.fyi/post/how-we-found-and-fixed-a-filesystem-corruption-bug.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Owen Jacobson
- Published: Feb 15, 2017
- Type: Incident / postmortem

An aborted Cedar-14 stack-image upgrade led container tools to truncate and overwrite a disk image while it was mounted as the read-only / filesystem for running dynos. Because direct writes bypassed filesystem caching and consistency mechanisms, reads alternated between cached pages from the old image and missing data from the truncated file, partially corrupting newly loaded programs. Applications consequently experienced elevated segmentation faults and other hard crashes for about five hours; a virtual-machine reproduction using an ext4 loop-mounted image and sleep reliably produced the same failure. Engineers migrated affected dynos to fresh hosts, then changed host-side permissions so installed filesystem images are read-only, and retested the conditions that triggered the incident.


### [Pulling the Thread on Kafka’s Compacted Topics](https://yomu.fyi/post/pulling-the-thread-on-kafka-s-compacted-topics.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Tom Crayford
- Published: Jan 11, 2017
- Type: Incident / postmortem

Apache Kafka’s log cleaner for compacted topics could crash silently when its offset map lacked enough space to hold a full log segment. Because the broker continued running without that internal thread, superseded data accumulated, threatening node and potentially cluster stability. The post explains compaction’s two phases: building an offset map for newer messages, then rewriting segments while retaining the latest offset for each key, with the latest segment excluded from the map. Kafka was changed to remove the crashing assertion and track partial segment loads so cleaning can recover without fitting an entire segment in memory. The roughly 100-line patch shipped in Kafka 0.10.1.1, which the post identifies as Heroku’s default version, and prevents the cleaner from dying silently.


### [Round-robin in Distributed Systems](https://yomu.fyi/post/round-robin-in-distributed-systems.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Gao Chao
- Published: Sep 27, 2016
- Type: Explainer

Building client-side load balancing for Grab's Common Data Service prompted a move from AWS Elastic Load Balancers to DNS discovery due to persistent connection issues and unpredictable scaling events. After patching an open-source library that failed to rotate IP sequences properly, the author evaluated different Go patterns for round-robin routing. A mutex-protected array counter provides the simplest model for basic retrieval, though adding mutations requires careful lock coordination. Alternatively, a dedicated balancer goroutine receiving requests over nested channels enables explicit operation timeouts and centralized event handling at the cost of higher code complexity and channel creation overhead. The author recommends the mutex approach for resource fetching and the goroutine-based design for workload balancing.


### [Grab You Some Post-Mortem Reports](https://yomu.fyi/post/grab-you-some-post-mortem-reports.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Lian Yuanlin
- Published: Feb 4, 2016
- Type: Problem & solution

Grab describes how its service-oriented architecture makes cross-team production debugging difficult because engineers may lack familiarity with other teams’ code and architecture. Post-mortem reports are presented as the glue for shared understanding, but a sample incident entry lacks service context, bug detail, impact, chronology, and investigation steps. The proposed standard has four requirements—Chronology, Context, Empowerment, and Solutions—covering timelines, service internals, blameless educational diagnosis, and remedies across People, Product, and Process. A template includes initial symptoms, a timestamped timeline, log-based investigation with code evidence, temporary and permanent fixes, and improvement actions, while peer review by another team is required before finalization.


### [Stuff Goes Bad](https://yomu.fyi/post/stuff-goes-bad.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Fred Hebert
- Published: Sep 17, 2014
- Type: Announcement

The Heroku Routing team describes a manual created to preserve hard-earned operational knowledge from developing and maintaining Erlang systems at scale. That knowledge had typically been passed informally through incident simulations, code reviews, and similar practices, leaving no persistent record for new team members. The manual presents itself as a guide to being an “Erlang medic” during failures, combining tips, tricks, code snippets, and practices for understanding failure sources and debugging running production systems. It is available as a free PDF under a Creative Commons License at erlang-in-anger.com, and was released in time for the Chicago Erlang conference, where Routing team members and Erlang community participants could discuss real-world applications.


[Newer posts](https://yomu.fyi/topic/reliability/page/14.md)