# Scalability
> 86 posts about Scalability, summarised, each linking to the original.

## Articles

### [Pharos - Searching Nearby Drivers on Road Network at Scale](https://yomu.fyi/post/pharos-searching-nearby-drivers-on-road-network-at-scale.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Hao Wu
- Published: Dec 22, 2020

Ride-hailing allocation requires identifying nearby drivers based on actual routing distance and estimated time of arrival rather than straight-line haversine distance. Grab developed Pharos, a distributed in-memory spatial microservice that evaluates road networks to match fast-moving drivers with pick-up requests. The system partitions OpenStreetMap road graphs by city and vehicle vertical while indexing driver positions and edge-based nodes using Adaptive Radix Trees. To locate candidates, Pharos projects driver coordinates onto road segments as phantom nodes and runs Incremental Network Expansion to compute isochrone reaches. Read-write concurrency is managed by switching tree root snapshots, allowing concurrent reads while updates produce new roots for subsequent lookups.


### [Democratising Fare Storage at Scale Using Event Sourcing](https://yomu.fyi/post/democratising-fare-storage-at-scale-using-event-sourcing.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Sourabh Suman
- Published: Nov 23, 2020

Grab's legacy system stored booking and fare details in a single relational table, creating a bloated booking entity that tracked only the latest fare state and hindered rapid feature iteration. To resolve scalability, stability, and debugging challenges across millions of daily bookings, the team developed Fare Storage using the Event Sourcing pattern. The new architecture persists all fare modification events chronologically in DynamoDB, backed by a cache for eventually consistent reads and message streaming for downstream processing. The platform employs optimistic locking with versioning to manage concurrent updates, enforces idempotency through client-generated transaction UUIDs, and delegates metadata serialization to an SDK to prevent storage API changes.


### [Developing Zoom Marketplace Apps w/ ngrok](https://yomu.fyi/post/developing-zoom-marketplace-apps-w-ngrok.md)
- Company: [Zoom](https://yomu.fyi/company/zoom.md)
- Author: Tim Slagle
- Published: Feb 14, 2020

Developers frequently use ngrok to establish fast, introspectable tunnels to localhost when building Zoom Marketplace applications instead of configuring complex reverse proxies like NGINX or Apache. However, using basic ngrok tunnels in production exposes systems to short URL expiration windows, single points of failure, and scalability bottlenecks. To maintain application availability, developers can purchase an ngrok license to secure a service-level agreement and support. Additionally, teams should reserve dedicated subdomains rather than using auto-generated endpoints, run ngrok as a monitored background service on a cloud provider or data center, and place a load balancer in front of the tunnel to handle heavy traffic.


### [How We Prevented App Performance Degradation from Sudden Ride Demand Spikes](https://yomu.fyi/post/how-we-prevented-app-performance-degradation-from-sudden-ride-demand-s.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Corey Scott
- Published: Jan 8, 2020

Grab experienced severe system strain when sudden localized spikes in ride demand, triggered by events like heavy rain or concert dismissals, coincided with driver shortages. These localized bursts overloaded the platform and degraded the experience for users outside the affected areas. To mitigate this, engineers created the Spampede filter, a circuit-breaker mechanism placed at the start of the booking pipeline. The filter converts pickup locations into Geohash Integer buckets and partitions time using Unix timestamps, tracking unfulfilled requests in Redis with atomic increments and time-to-live expirations. When unallocated requests exceed configured thresholds within a specific bucket, the system immediately short-circuits new incoming bookings to protect overall platform stability.


### [Griffin, an Anti-fraud Risk Rule Engine Making Billions of Predictions Daily](https://yomu.fyi/post/griffin-an-anti-fraud-risk-rule-engine-making-billions-of-predictions.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Muqi Li
- Published: Oct 28, 2019

Grab's Trust/Identity/Safety team built Griffin, an in-house anti-fraud risk rule engine designed to process billions of daily predictions across multiple business verticals. Initially, Grab managed fraud rules directly within backend service code, but escalating rule complexity, tight rule interdependencies, and translation gaps between data scientists and developers caused deployment delays and misfiring errors. To overcome the limitations and steep learning curves of third-party engines like Drools, the team separated the workflow into data orchestration and rule-based prediction. Griffin enables analysts and data scientists to author Python-based rules directly via a web portal and reload updated logic into memory without manual developer intervention. By eliminating I/O during rule evaluation and leveraging Gunicorn multi-processing, the engine handles over 100,000 queries per second at peak on six EC2 instances with single-prediction latencies under six milliseconds.


### [No More Forgetting to Input ERP Charges - Hello Automated ERP!](https://yomu.fyi/post/no-more-forgetting-to-input-erp-charges-hello-automated-erp.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Garvee Garg
- Published: Jul 31, 2019

Grab launched an automated Electronic Road Pricing (ERP) fare calculation feature in Singapore to eliminate the need for driver-partners to manually track gantries and enter toll charges. Because Singapore gantries frequently adjust fares based on time and road conditions, manual entry often caused driver errors and revenue loss. Grab solved this by mapping precise geographical coordinates for every toll gate using satellite imagery and open data, matching frequent driver GPS pings against road layers and gantry locations. The engineering and operations teams also built an internal ERP Workflow tool to map ride trajectories and resolve driver dispute feedback within an average of one day. Following its rollout in Singapore, Grab began testing and planning regional expansion to Indonesia, Thailand, Malaysia, and the Philippines.


### [How We Built a Logging Stack at Grab](https://yomu.fyi/post/how-we-built-a-logging-stack-at-grab.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Daniel Kasen
- Published: Jul 31, 2019

Grab needed a scalable logging platform to replace slow, fragmented systems that hindered debugging across their growing service fleet. Generating 25TB of daily logs, the team built a horizontally scalable Elasticsearch cluster configured via Ansible and monitored with Datadog. Although the initial proof of concept assigned all node roles (ingest, coordinator, master, and data) to every machine, operating at scale introduced major challenges with JVM heap exhaustion and cluster stability. The team resolved memory pressure and performance bottlenecks by tuning circuit breakers, lowering field data cache limits, adjusting shard allocations based on segment memory, and disabling translog compression during shard transfers.


### [Catwalk: Serving Machine Learning Models at Scale](https://yomu.fyi/post/catwalk-serving-machine-learning-models-at-scale.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Nutdanai Phansooksai
- Published: Jul 2, 2019

As machine learning adoption expanded at Grab, individual teams created fragmented model serving solutions that duplicated engineering effort and required data scientists to handle underlying infrastructure. To resolve these inefficiencies, Grab developed Catwalk, a self-service machine learning model serving platform. The system runs TensorFlow Serving containers across a managed Kubernetes cluster integrated with Grab's observability stack. Data scientists deploy or update models simply by saving files using the tf.saved\_model API to dedicated Amazon S3 buckets, while Kubernetes automates orchestration, ingress routing, and pod autoscaling. Catwalk abstracts server management away from data scientists, shortens deployment timelines, and provides high availability during model version rollouts.


### [Designing Resilient Systems Beyond Retries (Part 2): Bulkheading, Load Balancing, and Fallbacks](https://yomu.fyi/post/designing-resilient-systems-beyond-retries-part-2-bulkheading-load-bal.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Michael Cartmell
- Published: Mar 25, 2019

Software systems require mechanisms beyond retries to maintain resilience during downstream outages and high traffic. Bulkheading isolates failures across infrastructure, processes, thread pools, and connection limits, preventing a single failing component from degrading an entire system. Load balancing distributes traffic across backend pools via proxies, client-side libraries, lookaside services, or sidecars, often pairing with health checks to eliminate single points of failure. When operations fail unrecoverably, fallback strategies like silent failures, local defaults, stale cache reads, and dedicated backup services enable graceful degradation. Organizations like Grab implement these approaches using internal client-side load balancers backed by etcd, cache fallbacks in microservice frameworks, and redundant core backup services.


### [Designing Resilient Systems Beyond Retries (Part 1): Rate-Limiting](https://yomu.fyi/post/designing-resilient-systems-beyond-retries-part-1-rate-limiting.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Michael Cartmell
- Published: Mar 20, 2019

Distributed systems that rely exclusively on retries and circuit breakers face severe failure risks, including retry storms and reliance on client-side configuration accuracy. Implementing server-side rate limiting serves as a critical defensive layer to safeguard services across evolving architectures. Throttling thresholds can be layered across per-client, per-endpoint, and server-wide granularities using algorithms such as leaky bucket or sliding windows. While local instance-level limits fail when downstream bottlenecks like databases saturate under horizontal scaling, global rate limiting coordinates traffic enforcement across entire service pools. Centralized rate limiters require asynchronous communication and fallback mechanisms to avoid becoming single points of failure or adding request path latency.


### [Recipe for Building a Widget: How We Helped to “Peak-Shift” Demand by Helping Passengers Understand Travel Trends](https://yomu.fyi/post/recipe-for-building-a-widget-how-we-helped-to-peak-shift-demand-by-hel.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Lara PuReum Yim
- Published: Mar 7, 2019

Transport demand spikes during regular commuting hours often outpace driver availability, resulting in passenger wait times and fare surges. To mitigate these imbalances, Grab created the Travel Trends Widget for its mobile feed to redistribute ride requests toward off-peak windows. The widget uses machine learning forecasting to present historical supply-demand patterns alongside pricing trends for the upcoming two hours. To handle anticipated high query rates across millions of database entries, engineers periodically load precomputed trend data into an in-memory data structure rather than querying the database per request. The feature rolled out to feeds in Singapore and Jakarta within four weeks of initial development.


### [How We Simplified Our Data Ingestion & Transformation Process](https://yomu.fyi/post/how-we-simplified-our-data-ingestion-transformation-process.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Yichao Wang
- Published: Mar 3, 2019

Grab evolved its real-time data ingestion pipeline after an initial architecture built on Spark Streaming and Python encountered operational complexity, node failures, and data loss from S3 eventual consistency. Because the streaming workload primarily handled event partitioning and ORC file generation, the team consolidated these tasks directly into an existing Golang processing service. They implemented sharded concurrent maps for high-throughput partitioning and optimized heap allocations to resolve memory bottlenecks. This refactor removed intermediate Avro conversions and intermediate storage hops. The simplified Go pipeline eliminated data loss and reduced processing lag from up to 13 minutes down to approximately 1 minute.


### [Understanding Supply & Demand in Ride-hailing Through the Lens of Data](https://yomu.fyi/post/understanding-supply-demand-in-ride-hailing-through-the-lens-of-data.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Aayush Garg
- Published: Feb 20, 2019

Grab measures ride-hailing supply and demand across space and time to resolve geo-temporal allocation mismatches between moving drivers and ride-seeking passengers. The analytics pipeline defines supply as idle online drivers and demand as passengers checking fares within brief time slots, aggregating locations into geohashes. Each driver is mapped across neighbouring demand units and inversely weighted by straight-line distance, which yields the effective supply, supply-demand ratio, and supply-demand difference for each geographic polygon. Grab uses these aggregated metrics to identify marketplace imbalances, deploying driver heatmaps to shift excess supply and passenger travel trend widgets to defer time-insensitive ride requests.


### [A Lean and Scalable Data Pipeline to Capture Large Scale Events and Support Experimentation Platform](https://yomu.fyi/post/a-lean-and-scalable-data-pipeline-to-capture-large-scale-events-and-su.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Oscar Cassetti
- Published: Jan 16, 2019

Controlled online experimentation across diverse product verticals requires tracking interactions across systems to prevent local optimizations from causing global degradation. Grab built a batch data pipeline to capture, ingest, and process petabytes of event data to support its experimentation platform and analytics stakeholders. The architecture loads ingested event data from Amazon S3, transforms and sorts it, and writes partitioned output back to S3 with metadata registered in Apache Hive. Using Apache Spark on AWS Elastic MapReduce with Apache Airflow for orchestration, the system handles roughly 400,000 incoming events per second. The data is partitioned by event type and ingestion time and stored in Apache ORC format to streamline query workloads and reduce retrieval overhead.


### [Designing Resilient Systems: Circuit Breakers or Retries? (Part 2)](https://yomu.fyi/post/designing-resilient-systems-circuit-breakers-or-retries-part-2.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Corey Scott
- Published: Jan 8, 2019

Retries enable software systems to recover from transient upstream failures by automatically repeating unsuccessful requests. While retrying increases the chance of request completion across multi-host setups, it consumes additional CPU and time without inherently tracking host health. Applications must selectively retry errors with a likelihood of success, such as 500 and 503 status codes, while avoiding client-side failures like 400 or 401. To manage distributed systems safely, retries require idempotent operations or cryptographic nonces, along with backoff and jitter to prevent request stampedes. Tuning retry counts, timeouts, and delays is critical to cap the worst-case consumer response time.


### [Querying Big Data in Real-time with Presto & Grab's TalariaDB](https://yomu.fyi/post/querying-big-data-in-real-time-with-presto-grab-s-talariadb.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Roman Atachiants
- Published: Jan 2, 2019

Grab developed TalariaDB to support real-time SQL querying over high-velocity event streams while maintaining predictable sub-second latencies and low infrastructure costs. The distributed time-series store retains only the most recent hour of data and integrates directly with Presto via its PrestoThriftService interface. Internally, TalariaDB uses the Go-based Badger key-value store to maintain an in-memory key index of metric names and timestamps while mapping columnar event payloads directly to disk. Ingestion occurs by processing pre-partitioned event batches written to Amazon S3 via SQS notifications. By combining a zero-copy decoder with parallel split evaluation across gossiping cluster nodes, the architecture scales horizontally while serving millions of events per second.


### [Reliable and Scalable Feature Toggles and A/B Testing SDK at Grab](https://yomu.fyi/post/reliable-and-scalable-feature-toggles-and-a-b-testing-sdk-at-grab.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Roman Atachiants
- Published: Nov 2, 2018

Grab previously managed experiments using custom service-level code and a toggling library that queried a shared Redis instance, creating latency risks and a single point of failure across backend microservices. To achieve reliable, sub-microsecond feature evaluations, the team designed a Go SDK that resolves rollouts and A/B tests entirely in memory without runtime network I/O. Backend services periodically poll JSON-defined configuration schemas stored in Amazon S3 through a Universal Configuration Manager. The SDK evaluates contextual attributes called facets locally and pushes decision telemetry asynchronously to an S3 and Presto data lake. This architecture allows engineering and product teams to gate deployments and run server-side experiments safely without service disruption.


### [How We Designed the Quotas Microservice to Prevent Resource Abuse](https://yomu.fyi/post/how-we-designed-the-quotas-microservice-to-prevent-resource-abuse.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Jim Zhan
- Published: Aug 10, 2018

As Grab migrated from a monolith to hundreds of microservices, managing global rate limiting became essential to prevent cascading failures and resource exhaustion. To avoid putting a rate limiting service on the critical path of every API call, Grab built Quotas, an asynchronous rate limiting system. Client services use a lightweight SDK and middleware to read rate limiting decisions from local in-memory caches and stream usage metrics asynchronously via Apache Kafka. The Quotas service aggregates usage data locally, flushes stats to Redis periodically, and publishes updated rate limiting decisions back over Kafka topics. In production, Quotas successfully handles 200k peak transactions per second with decision enforcement delays capped at 200 milliseconds.


### [Grab Senior Data Scientist Liuqin Yang Wins Beale-Orchard-Hays Prize](https://yomu.fyi/post/grab-senior-data-scientist-liuqin-yang-wins-beale-orchard-hays-prize.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Yang Liuqin
- Published: Jul 20, 2018

Grab Senior Data Scientist Dr. Liuqin Yang, Professor Defeng Sun, and Professor Kim-Chuan Toh received the 2018 Beale-Orchard-Hays Prize for their research paper introducing SDPNAL+. The software employs a majorised semismooth Newton-CG augmented Lagrangian method to solve large-scale semidefinite programming problems with nonnegative constraints. While traditional methods struggled beyond matrix dimensions of 2,000 and 5,000 constraints, SDPNAL+ successfully scales to matrix dimensions of 9,261 and over 12 million constraints. In benchmark testing, the software solved a problem on a desktop PC in 1.5 hours that required 122 hours on a 56-core CPU and 128-GPU cluster using a traditional solver. Grab implements these optimisation techniques to accelerate its passenger-driver allocation algorithms by hundreds of times.


### [Building Grab’s Experimentation Platform](https://yomu.fyi/post/building-grab-s-experimentation-platform.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Abeesh Thomas
- Published: Jul 13, 2018

Grab built its internal Experimentation Platform (ExP) to replace a manual, expensive testing process that required bespoke meetings, custom logging pipelines, and service modifications for each experiment. ExP provides a unified infrastructure featuring a centralized management UI, automated real-time data streaming to S3, and SDKs for Android, iOS, and Go. The platform leverages JSON-based experiment definitions delivered through dynamic configuration management, enabling client-side evaluation without costly network calls. It addresses marketplace network effects and inter-experiment interference through mechanisms such as geo-temporal segmentation and domain-layer models. The platform has scaled to run approximately 25 concurrent experiments while computing roughly 2,500 metrics and 50,000 experiment-metric combinations daily.


[Newer posts](https://yomu.fyi/topic/scalability/page/3.md) · [Older posts](https://yomu.fyi/topic/scalability/page/5.md)