# Reliability > 62 posts about Reliability, summarised, each linking to the original. ## Articles ### [Democratising Fare Storage at Scale Using Event Sourcing](https://yomu.fyi/post/democratising-fare-storage-at-scale-using-event-sourcing.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Sourabh Suman - Published: Nov 23, 2020 Grab's legacy system stored booking and fare details in a single relational table, creating a bloated booking entity that tracked only the latest fare state and hindered rapid feature iteration. To resolve scalability, stability, and debugging challenges across millions of daily bookings, the team developed Fare Storage using the Event Sourcing pattern. The new architecture persists all fare modification events chronologically in DynamoDB, backed by a cache for eventually consistent reads and message streaming for downstream processing. The platform employs optimistic locking with versioning to manage concurrent updates, enforces idempotency through client-generated transaction UUIDs, and delegates metadata serialization to an SDK to prevent storage API changes. ### [Uncovering the Truth Behind Lua and Redis Data Consistency](https://yomu.fyi/post/uncovering-the-truth-behind-lua-and-redis-data-consistency.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Allen Wang - Published: Sep 7, 2020 Grab experienced replica CPU usage spikes following service deployments in their master/replica Redis cluster, which caused failovers to spike to 100% CPU. Investigation revealed that a post-deployment Lua monitor script executed separately on both nodes and relied on non-deterministic HGETALL key ordering. Redis encodes hash objects as either ziplists or hashtables, and restoring from an RDB snapshot initializes small hashes as ziplists even if the master previously converted them to hashtables. This encoding discrepancy caused key ordering to diverge, preventing secondary data from deleting correctly and bloating dataset sizes. Grab resolved the issue by sorting the outputs of HKEYS and HGETALL within the Lua script to guarantee deterministic execution across nodes. ### [Developing Zoom Marketplace Apps w/ ngrok](https://yomu.fyi/post/developing-zoom-marketplace-apps-w-ngrok.md) - Company: [Zoom](https://yomu.fyi/company/zoom.md) - Author: Tim Slagle - Published: Feb 14, 2020 Developers frequently use ngrok to establish fast, introspectable tunnels to localhost when building Zoom Marketplace applications instead of configuring complex reverse proxies like NGINX or Apache. However, using basic ngrok tunnels in production exposes systems to short URL expiration windows, single points of failure, and scalability bottlenecks. To maintain application availability, developers can purchase an ngrok license to secure a service-level agreement and support. Additionally, teams should reserve dedicated subdomains rather than using auto-generated endpoints, run ngrok as a monitored background service on a cloud provider or data center, and place a load balancer in front of the tunnel to handle heavy traffic. ### [How We Prevented App Performance Degradation from Sudden Ride Demand Spikes](https://yomu.fyi/post/how-we-prevented-app-performance-degradation-from-sudden-ride-demand-s.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Corey Scott - Published: Jan 8, 2020 Grab experienced severe system strain when sudden localized spikes in ride demand, triggered by events like heavy rain or concert dismissals, coincided with driver shortages. These localized bursts overloaded the platform and degraded the experience for users outside the affected areas. To mitigate this, engineers created the Spampede filter, a circuit-breaker mechanism placed at the start of the booking pipeline. The filter converts pickup locations into Geohash Integer buckets and partitions time using Unix timestamps, tracking unfulfilled requests in Redis with atomic increments and time-to-live expirations. When unallocated requests exceed configured thresholds within a specific bucket, the system immediately short-circuits new incoming bookings to protect overall platform stability. ### [Preventing Pipeline Calls from Crashing Redis Clusters](https://yomu.fyi/post/preventing-pipeline-calls-from-crashing-redis-clusters.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Michael Cartmell - Published: May 5, 2019 A single Redis slave node failure caused Grab's Apollo booking service to suffer an over 95 percent call failure rate for one minute despite running a three-shard cluster with two replicas per partition. Investigation revealed that the service configured Go-Redis to route all read queries exclusively to slave nodes to offload master CPU usage. When the slave node dropped offline, batched HMGET pipeline calls failed completely because the client wrapper treated a single command failure as a failure of the entire pipeline. Furthermore, the Go-Redis client cached cluster topology and only lazily refreshed state every sixty seconds, continuing to direct traffic to the dead replica until the timer expired. Grab addressed this risk by recommending dedicated pipeline clients configured with latency-based routing to allow reads to fall back to responsive master nodes. ### [How We Harnessed the Wisdom of Crowds to Improve Restaurant Location Accuracy](https://yomu.fyi/post/how-we-harnessed-the-wisdom-of-crowds-to-improve-restaurant-location-a.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Pravin Kakar - Published: Apr 2, 2019 Grab discovered that abnormally short driver wait times often indicated restaurants registered at incorrect coordinates due to moves or onboarding errors. To fix this, Grab used driver-partner GPS pings, timestamps, and order status updates to infer true food collection locations. The system cleans the data by filtering low-quality GPS pings and isolating the longest temporal streak a driver spends within a predefined radius of the venue. Clusters of inferred pick-up points are then ranked by order volume, the proportion of off-target pick-ups, and median distance errors before routing to mapping operations for verification. This periodic correction workflow achieved a fivefold reduction in order cancellations caused by unfound merchant locations. ### [Designing Resilient Systems Beyond Retries (Part 3): Architecture Patterns and Chaos Engineering](https://yomu.fyi/post/designing-resilient-systems-beyond-retries-part-3-architecture-pattern.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Michael Cartmell - Published: Mar 27, 2019 Building resilient systems requires architectural safeguards and proactive testing beyond basic retries and circuit breakers. Architectural patterns such as idempotency keys enable safe retries without creating inconsistent state during failures. Asynchronous responses and deferrable work isolate services from downstream dependency latency and errors, though they can conflict with the fail-fast principle. To validate system behavior under stress, chaos engineering introduces intentional failures in production to test hypotheses against a defined steady state. Selectively adopting complementary patterns reduces failure points while avoiding unnecessary architectural complexity. ### [Designing Resilient Systems Beyond Retries (Part 2): Bulkheading, Load Balancing, and Fallbacks](https://yomu.fyi/post/designing-resilient-systems-beyond-retries-part-2-bulkheading-load-bal.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Michael Cartmell - Published: Mar 25, 2019 Software systems require mechanisms beyond retries to maintain resilience during downstream outages and high traffic. Bulkheading isolates failures across infrastructure, processes, thread pools, and connection limits, preventing a single failing component from degrading an entire system. Load balancing distributes traffic across backend pools via proxies, client-side libraries, lookaside services, or sidecars, often pairing with health checks to eliminate single points of failure. When operations fail unrecoverably, fallback strategies like silent failures, local defaults, stale cache reads, and dedicated backup services enable graceful degradation. Organizations like Grab implement these approaches using internal client-side load balancers backed by etcd, cache fallbacks in microservice frameworks, and redundant core backup services. ### [Designing Resilient Systems Beyond Retries (Part 1): Rate-Limiting](https://yomu.fyi/post/designing-resilient-systems-beyond-retries-part-1-rate-limiting.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Michael Cartmell - Published: Mar 20, 2019 Distributed systems that rely exclusively on retries and circuit breakers face severe failure risks, including retry storms and reliance on client-side configuration accuracy. Implementing server-side rate limiting serves as a critical defensive layer to safeguard services across evolving architectures. Throttling thresholds can be layered across per-client, per-endpoint, and server-wide granularities using algorithms such as leaky bucket or sliding windows. While local instance-level limits fail when downstream bottlenecks like databases saturate under horizontal scaling, global rate limiting coordinates traffic enforcement across entire service pools. Centralized rate limiters require asynchronous communication and fallback mechanisms to avoid becoming single points of failure or adding request path latency. ### [Context Deadlines and How to Set Them](https://yomu.fyi/post/context-deadlines-and-how-to-set-them.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Michael Cartmell - Published: Mar 11, 2019 Microservice architectures operating under heavy network traffic require robust timeout handling to prevent slow or failing dependencies from causing cascading failures across services. Naive, static timeout configurations across a call chain often cause downstream components to waste compute effort on requests that upstream callers have already abandoned. To establish predictable timeout thresholds, engineers can align limits with service-level latency percentiles, pairing P99 limits with median latency retry allowances. Go's context package improves upon static network timeouts by propagating request-scoped deadlines and cancellation signals across service boundaries. This distributed context ensures downstream servers recognize remaining time budgets and terminate unneeded processing immediately when parent deadlines expire or callers manually cancel requests. ### [Designing Resilient Systems: Circuit Breakers or Retries? (Part 2)](https://yomu.fyi/post/designing-resilient-systems-circuit-breakers-or-retries-part-2.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Corey Scott - Published: Jan 8, 2019 Retries enable software systems to recover from transient upstream failures by automatically repeating unsuccessful requests. While retrying increases the chance of request completion across multi-host setups, it consumes additional CPU and time without inherently tracking host health. Applications must selectively retry errors with a likelihood of success, such as 500 and 503 status codes, while avoiding client-side failures like 400 or 401. To manage distributed systems safely, retries require idempotent operations or cryptographic nonces, along with backoff and jitter to prevent request stampedes. Tuning retry counts, timeouts, and delays is critical to cap the worst-case consumer response time. ### [Designing Resilient Systems: Circuit Breakers or Retries? (Part 1)](https://yomu.fyi/post/designing-resilient-systems-circuit-breakers-or-retries-part-1.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Corey Scott - Published: Dec 21, 2018 Distributed architectures frequently encounter upstream failures triggered by networking issues, system overloads, resource starvation, and invalid deployments. Implementing software circuit breakers interposes a monitoring mechanism between components to halt requests when failure thresholds are met, giving struggling upstream dependencies time to recover. Circuit breakers save CPU, memory, and network resources by failing fast or routing execution through defined fallbacks such as cached data, alternate services, or approximation algorithms. Grab utilizes Hystrix-Go to manage upstream interactions and configure key thresholds for concurrency, timeouts, and error ratios. This approach protects downstream consumers from cascading latency while insulating upstream resources from excess traffic. ### [Orchestrating Chaos Using Grab's Experimentation Platform](https://yomu.fyi/post/orchestrating-chaos-using-grab-s-experimentation-platform.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Roman Atachiants - Published: Nov 23, 2018 Grab operates hundreds of microservices where failures in non-critical components can cause outages in critical user flows if fallback mechanisms are improperly configured. To validate system resilience, Grab built Chaos ExP by layering a chaos engineering SDK and dedicated web UI on top of its existing Experimentation Platform. Integrated directly into the Grab-Kit server middleware, the framework intercepts incoming requests and evaluates whether to inject failures using local variable resolution. Supported failure primitives include latency, errors, panics, rate throttling, and resource leaks to test dependent services. Combining chaos testing with experimentation telemetry enables engineers to correlate injected infrastructure disruptions with business metric impacts. ### [Reliable and Scalable Feature Toggles and A/B Testing SDK at Grab](https://yomu.fyi/post/reliable-and-scalable-feature-toggles-and-a-b-testing-sdk-at-grab.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Roman Atachiants - Published: Nov 2, 2018 Grab previously managed experiments using custom service-level code and a toggling library that queried a shared Redis instance, creating latency risks and a single point of failure across backend microservices. To achieve reliable, sub-microsecond feature evaluations, the team designed a Go SDK that resolves rollouts and A/B tests entirely in memory without runtime network I/O. Backend services periodically poll JSON-defined configuration schemas stored in Amazon S3 through a Universal Configuration Manager. The SDK evaluates contextual attributes called facets locally and pushes decision telemetry asynchronously to an S3 and Presto data lake. This architecture allows engineering and product teams to gate deployments and run server-side experiments safely without service disruption. ### [Deep Dive into Database Timeouts in Rails](https://yomu.fyi/post/deep-dive-into-database-timeouts-in-rails.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Jia Hao Goh - Published: Jan 29, 2018 Following a production outage where a database failover caused a Ruby on Rails application to exhaust its Puma server threads, an investigation was conducted to understand how ActiveRecord and MySQL timeout settings behave. A reproduction environment using Docker, Puma, and Toxiproxy replicated how hanging requests to a failing database consume all available server threads, ultimately starving unrelated endpoints. The analysis breaks down ActiveRecord connection pooling mechanics alongside underlying mysql2 and libmysqlclient settings, specifically checkout\_timeout, connect\_timeout, and read\_timeout. Testing confirmed how existing and new TCP connections transition through socket states during network interruptions while waiting on configured timeout intervals. ### [GrabShare at the Intelligent Transportation Engineering Conference](https://yomu.fyi/post/grabshare-at-the-intelligent-transportation-engineering-conference.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Dominic Widdows - Published: Dec 13, 2017 Grab presented a technical paper on the construction of its real-time ridesharing service, GrabShare, at the Intelligent Transportation Engineering Conference in Singapore. The platform pairs passengers heading along similar routes with drivers immediately while handling network drops, volatile supply and demand, and heavy traffic conditions in Southeast Asian cities. To deliver accurate pairings, the scheduling system generates and filters through hundreds of travel time estimates for each candidate match before finalizing an itinerary. Operational teams on the ground evaluate complaints about poor matches, enabling engineers to refine the online matching systems. Over the course of one month, the service cut more than 4.5 million kilometers of driving distance and brought in over 100,000 new users within two weeks. ### [Come and #hackallthethings at Grab](https://yomu.fyi/post/come-and-hackallthethings-at-grab.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Grab Engineering - Published: Jul 11, 2017 Grab has officially launched a public bug bounty program in partnership with HackerOne to strengthen the security of its platform. This rollout follows a private bounty initiative operated over the previous year, during which the organization worked with over 350 security researchers and resolved nearly 200 awarded bug reports. The new public program invites external researchers to scrutinize Grab's code for critical flaws, including remote code execution, SQL injections, and exportable cross-site scripting vulnerabilities. To support ethical and responsible disclosure, Grab offers payouts reaching up to $10,000 per valid vulnerability report based on severity and impact. ### [How to Go from a Quick Idea to an Essential Feature in Four Steps](https://yomu.fyi/post/how-to-go-from-a-quick-idea-to-an-essential-feature-in-four-steps.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Da Huang - Published: May 16, 2017 Grab engineered an in-app messaging platform, GrabChat, to help drivers and passengers coordinate pickups across Southeast Asian markets characterized by weak 2G connectivity and high packet loss. The team developed an in-house TCP messaging architecture consisting of a TCP gateway named Gundam and a message dispatcher named Hermes connected to internal backend services over HTTPS. To protect backend server resources from resend loops during poor connection states, the communication protocol adopts a "server only push once" model that delegates retry handling to the client. Data science evaluations using a pre-trained cancellation prediction model confirmed that GrabChat adoption correlated with reduced booking cancellations. Following early usage feedback, the team further iterated on the feature by introducing pre-written message templates to reduce driver distraction on the road. ### [Troubleshooting Unusual AWS ELB 5XX Error](https://yomu.fyi/post/troubleshooting-unusual-aws-elb-5xx-error.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Dharmarth Shah - Published: May 10, 2017 Grab experienced intermittent HTTP 5XX alerts when its Gothena service sent driver location updates to the Astrolabe service through an AWS Elastic Load Balancer (ELB). CloudWatch metrics revealed that requests were failing to reach healthy backend instances because of an uneven load distribution favoring a single ELB node in one Availability Zone. The team verified that Route 53 was properly using Alias records and ruled out OS-level DNS caching since Linux does not cache DNS queries by default. Connection inspection with netstat across multiple Go services confirmed a heavily skewed distribution of connections toward specific ELB IP addresses. Comparative tests with cURL, tcpdump, Go, Python, and Ruby in an isolated environment demonstrated that Go reused connections across requests while other runtimes opened new connections per request. ### [A Key Expired in Redis, You Won't Believe What Happened Next](https://yomu.fyi/post/a-key-expired-in-redis-you-won-t-believe-what-happened-next.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Karan Kamath - Published: Mar 27, 2017 Grab experienced an issue where its Unicorn API served stale data for up to 45 to 60 minutes despite expected cache invalidation times totaling around 11 minutes. The setup utilized ElastiCache Redis 2.x configured with a single master node for writes and two read-only slaves handling reads. Investigation revealed that in Redis 2.x, slave nodes do not expire keys on their own and only delete them upon receiving an explicit DEL command from the master. Because the master only actively checks and deletes 200 random keys per second, clearing expired keys across roughly 5.6 million cached items mathematically required over 110 hours, resulting in slaves serving expired data. [Newer posts](https://yomu.fyi/topic/reliability/page/2.md) ยท [Older posts](https://yomu.fyi/topic/reliability/page/4.md)