---
title: "Scalability"
description: "379 posts about Scalability, summarised, each linking to the original."
---

# Scalability
> 379 posts about Scalability, summarised, each linking to the original.

## Articles

### [A Lean and Scalable Data Pipeline to Capture Large Scale Events and Support Experimentation Platform](https://yomu.fyi/post/a-lean-and-scalable-data-pipeline-to-capture-large-scale-events-and-su.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Oscar Cassetti
- Published: Jan 16, 2019
- Type: Problem & solution

Controlled online experimentation across diverse product verticals requires tracking interactions across systems to prevent local optimizations from causing global degradation. Grab built a batch data pipeline to capture, ingest, and process petabytes of event data to support its experimentation platform and analytics stakeholders. The architecture loads ingested event data from Amazon S3, transforms and sorts it, and writes partitioned output back to S3 with metadata registered in Apache Hive. Using Apache Spark on AWS Elastic MapReduce with Apache Airflow for orchestration, the system handles roughly 400,000 incoming events per second. The data is partitioned by event type and ingestion time and stored in Apache ORC format to streamline query workloads and reduce retrieval overhead.


### [Designing Resilient Systems: Circuit Breakers or Retries? (Part 2)](https://yomu.fyi/post/designing-resilient-systems-circuit-breakers-or-retries-part-2.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Corey Scott
- Published: Jan 8, 2019
- Type: Explainer

Retries enable software systems to recover from transient upstream failures by automatically repeating unsuccessful requests. While retrying increases the chance of request completion across multi-host setups, it consumes additional CPU and time without inherently tracking host health. Applications must selectively retry errors with a likelihood of success, such as 500 and 503 status codes, while avoiding client-side failures like 400 or 401. To manage distributed systems safely, retries require idempotent operations or cryptographic nonces, along with backoff and jitter to prevent request stampedes. Tuning retry counts, timeouts, and delays is critical to cap the worst-case consumer response time.


### [Querying Big Data in Real-time with Presto & Grab's TalariaDB](https://yomu.fyi/post/querying-big-data-in-real-time-with-presto-grab-s-talariadb.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Roman Atachiants
- Published: Jan 2, 2019
- Type: Problem & solution

Grab developed TalariaDB to support real-time SQL querying over high-velocity event streams while maintaining predictable sub-second latencies and low infrastructure costs. The distributed time-series store retains only the most recent hour of data and integrates directly with Presto via its PrestoThriftService interface. Internally, TalariaDB uses the Go-based Badger key-value store to maintain an in-memory key index of metric names and timestamps while mapping columnar event payloads directly to disk. Ingestion occurs by processing pre-partitioned event batches written to Amazon S3 via SQS notifications. By combining a zero-copy decoder with parallel split evaluation across gossiping cluster nodes, the architecture scales horizontally while serving millions of events per second.


### [Preparing Shopify for Black Friday and Cyber Monday](https://yomu.fyi/post/preparing-shopify-for-black-friday-and-cyber-monday.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: Camilo Lopez
- Published: Dec 21, 2018
- Type: Explainer

Shopify’s Production Engineering team prepares its platform and 600,000+ merchants for Black Friday and Cyber Monday traffic without interrupting service. Preparation combines controlled fault injection “game days,” synthetic load tests, and staged freezes: a feature freeze begins several weeks ahead, followed by a code freeze allowing only critical fixes during the event. The tests probe dependencies such as shipping providers and payment gateways, including latency, fallback behavior, and responses to datastore failures. After the 2018 weekend, Shopify reported nearly 11,000 orders per minute, around 100,000 requests per second for extended periods, and overall metrics about 1.8 times 2017 levels. The account concludes that fault injection, load testing, and disaster role-play improved readiness, while third-party downtime remains Shopify’s responsibility to merchants and buyers.


### [How an Intern Released 3 Terabytes Worth of Storage Before BFCM](https://yomu.fyi/post/how-an-intern-released-3-terabytes-worth-of-storage-before-bfcm.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: Gurpreet Gill
- Published: Dec 14, 2018
- Type: Problem & solution

While preparing for Black Friday Cyber Monday (BFCM), Shopify’s Products team found that historical reporting records were causing MySQL to scan many InnoDB pages to return a single record. An intern built a Rails background job to delete the records safely, using Shopify’s job-iteration library to process small batches and an internal throttling enumerator to pause when database health or replication lag became a concern. Indexed fetch queries and cursor-based batching avoided repeated scans and timeouts, while the iterator deleted each batch. Run about two weeks before BFCM, the job deleted more than 5 billion records, peaked at approximately six million records per minute, and released three terabytes of storage. Afterward, the database scanned around three times fewer pages per returned record, improving resilience to flash-sale traffic and producing large cost savings.


### [Reliable and Scalable Feature Toggles and A/B Testing SDK at Grab](https://yomu.fyi/post/reliable-and-scalable-feature-toggles-and-a-b-testing-sdk-at-grab.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Roman Atachiants
- Published: Nov 2, 2018
- Type: Problem & solution

Grab previously managed experiments using custom service-level code and a toggling library that queried a shared Redis instance, creating latency risks and a single point of failure across backend microservices. To achieve reliable, sub-microsecond feature evaluations, the team designed a Go SDK that resolves rollouts and A/B tests entirely in memory without runtime network I/O. Backend services periodically poll JSON-defined configuration schemas stored in Amazon S3 through a Universal Configuration Manager. The SDK evaluates contextual attributes called facets locally and pushes decision telemetry asynchronously to an S3 and Presto data lake. This architecture allows engineering and product teams to gate deployments and run server-side experiments safely without service disruption.


### [How We Designed the Quotas Microservice to Prevent Resource Abuse](https://yomu.fyi/post/how-we-designed-the-quotas-microservice-to-prevent-resource-abuse.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Jim Zhan
- Published: Aug 10, 2018
- Type: Problem & solution

As Grab migrated from a monolith to hundreds of microservices, managing global rate limiting became essential to prevent cascading failures and resource exhaustion. To avoid putting a rate limiting service on the critical path of every API call, Grab built Quotas, an asynchronous rate limiting system. Client services use a lightweight SDK and middleware to read rate limiting decisions from local in-memory caches and stream usage metrics asynchronously via Apache Kafka. The Quotas service aggregates usage data locally, flushes stats to Redis periodically, and publishes updated rate limiting decisions back over Kafka topics. In production, Quotas successfully handles 200k peak transactions per second with decision enforcement delays capped at 200 milliseconds.


### [Grab Senior Data Scientist Liuqin Yang Wins Beale-Orchard-Hays Prize](https://yomu.fyi/post/grab-senior-data-scientist-liuqin-yang-wins-beale-orchard-hays-prize.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Yang Liuqin
- Published: Jul 20, 2018
- Type: Announcement

Grab Senior Data Scientist Dr. Liuqin Yang, Professor Defeng Sun, and Professor Kim-Chuan Toh received the 2018 Beale-Orchard-Hays Prize for their research paper introducing SDPNAL+. The software employs a majorised semismooth Newton-CG augmented Lagrangian method to solve large-scale semidefinite programming problems with nonnegative constraints. While traditional methods struggled beyond matrix dimensions of 2,000 and 5,000 constraints, SDPNAL+ successfully scales to matrix dimensions of 9,261 and over 12 million constraints. In benchmark testing, the software solved a problem on a desktop PC in 1.5 hours that required 122 hours on a 56-core CPU and 128-GPU cluster using a traditional solver. Grab implements these optimisation techniques to accelerate its passenger-driver allocation algorithms by hundreds of times.


### [Building Grab’s Experimentation Platform](https://yomu.fyi/post/building-grab-s-experimentation-platform.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Abeesh Thomas
- Published: Jul 13, 2018
- Type: Problem & solution

Grab built its internal Experimentation Platform (ExP) to replace a manual, expensive testing process that required bespoke meetings, custom logging pipelines, and service modifications for each experiment. ExP provides a unified infrastructure featuring a centralized management UI, automated real-time data streaming to S3, and SDKs for Android, iOS, and Go. The platform leverages JSON-based experiment definitions delivered through dynamic configuration management, enabling client-side evaluation without costly network calls. It addresses marketplace network effects and inter-experiment interference through mechanisms such as geo-temporal segmentation and domain-layer models. The platform has scaled to run approximately 25 concurrent experiments while computing roughly 2,500 metrics and 50,000 experiment-metric combinations daily.


### [A Pods Architecture To Allow Shopify To Scale](https://yomu.fyi/post/a-pods-architecture-to-allow-shopify-to-scale.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: Xavier Denis
- Published: Mar 2, 2018
- Type: Explainer

Shopify outgrew its ability to scale by buying larger database servers, so it introduced database sharding in 2015. Sharding improved performance and horizontal scalability but made actions dependent on every shard, increasing the risk of platform-wide unavailability. In 2016, Shopify reorganized its runtime around pods: each pod groups shops on isolated datastores, while shared workers, application servers, and load balancers are restricted to one pod at a time. Sorting Hat assigns each request to a pod and location, while every web request and delayed job is kept within one pod; paired data centers and Pod Mover support recovery without dropping requests or jobs. The design makes a pod the unit for scalability, resiliency, and disaster recovery, and simplifies evacuating an entire data center.


### [Scaling ipify to 30 Billion Requests and Beyond on Heroku](https://yomu.fyi/post/scaling-ipify-to-30-billion-requests-and-beyond-on-heroku.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Randall Degges
- Published: Jan 23, 2018
- Type: Problem & solution

ipify began as a free API that returns a caller’s public-facing IP address, created to help discover cloud instances’ external addresses without management APIs. Its initial implementation was a sub-50-Loc Node service deployed on one Heroku dyno for about $7/month; rising usage caused slow responses and 503s. Investigation showed roughly 10 rps, while one Node process and string manipulation around X-Forwarded-For limited throughput; clustering doubled capacity, and the implementation later used Go. Heroku supplied load balancing, automatic restarts, and horizontal dyno scaling as traffic expanded through bursty demand and malicious use; the operator chose manual capacity rather than paid autoscaling. The service surpassed 30B requests per month, routinely serving 2k–20k RPS with 1–20ms average response times at $150–$200 monthly; ownership later passed to whoisxmlapi.com, whose work includes an improved UI and more data endpoint.


### [Grabbing Growth: A Growth Hacking Story](https://yomu.fyi/post/grabbing-growth-a-growth-hacking-story.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Gaurav Sachdeva
- Published: Dec 8, 2017
- Type: Problem & solution

Grab established a dedicated Growth Hacking team within its Technology organization to pursue high-risk, niche initiatives and scale impact across 68 million regional users. Positioning the team within engineering allowed rapid A/B testing and simultaneous multi-market deployments driven by structured growth loops. The team prioritizes and evaluates all initiatives using a Growth Factor metric, calculated as the increase in rides divided by the increase in costs. To improve driver engagement, the team implemented a Spin-to-Win game based on B.F. Skinner's variable ratio reinforcement principles, delivering probabilistic monetary and merchandise rewards upon meeting daily ride thresholds. Ongoing regional experiments continue to evaluate metrics including driver acceptance, cancellation rates, and driver ratings.


### [Namespacing Actions for Redux](https://yomu.fyi/post/namespacing-actions-for-redux.md)
- Company: [Kickstarter](https://yomu.fyi/company/kickstarter.md)
- Author: Sarah Groff Hennigh-Palermo
- Published: Nov 20, 2017
- Type: Explainer

Kickstarter’s front-end team encountered a Redux action-namespacing problem when multiple instances of the same functional React component responded to actions intended for only one instance. They rejected local state because the application used functional components and redux-loop, which routes state updates and side effects through reducers, and they sought programmatic namespacing compatible with combineReducers and a low-surface-area component API. The team compared nested reducers, higher-order reducers, and a closure-based module pattern; the latter met the stated constraints with namespaced actions, wrapper components, reducers, and a namespace returned by a factory that defaults to a UUID. Namespaces must be preserved when reducers trigger further actions, including redux-loop results, and complex composition can lose the required binding. The approach has worked for most reusable and compound components, while possible performance costs from many reducers remain a future concern.


### [How We Scaled Our Cache and Got a Good Night's Sleep](https://yomu.fyi/post/how-we-scaled-our-cache-and-got-a-good-night-s-sleep.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Gao Chao
- Published: Jun 19, 2017
- Type: Problem & solution

Growing business load on the Common Data Service (CDS) created potential bottlenecks for its single-threaded Redis cache on ElastiCache, necessitating horizontal scaling for greater capacity and throughput. After ruling out master-slave replication and intermediate Twemproxy setups due to memory constraints and proxy I/O bottlenecks, the team implemented client-side sharding. Using an internal Go package for consistent hashing, CDS instances hash cache keys locally to determine the target shard. The implementation encapsulates hashing inside a thin \`ShardedCache\` wrapper sharing the original cache interface while supporting Ketama and custom hash functions. Deploying via double-writing cron jobs during off-peak hours reduced database read pressure and improved P99 latency.


### [Driving Southeast Asia Forward with AWS](https://yomu.fyi/post/driving-southeast-asia-forward-with-aws.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Arul Kumaravel
- Published: May 21, 2017
- Type: Explainer

Grab transitioned its transportation platform from a single Ruby on Rails monolith on Amazon EC2 and Amazon RDS MySQL to a microservices architecture hosted on Amazon Web Services. The platform processes multi-petabyte real-time data flows and hundreds of millions of GPS data points to match drivers with passengers and push proactive demand heat maps. Operational efficiency is sustained with fewer than ten full-time infrastructure engineers, supported by AWS managed services. The analytics backend also transitioned from MySQL to Amazon Redshift, eventually moving to an Amazon S3 data lake using Amazon EMR and Presto. These data-driven matching systems improved driver-passenger allocation rates by up to 30%.


### [Scaling Like a Boss with Presto](https://yomu.fyi/post/scaling-like-a-boss-with-presto.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Aneesh Chandra
- Published: May 1, 2017
- Type: Problem & solution

Grab experienced severe performance degradation, long queue times, and connection timeouts on its Amazon Redshift analytics cluster as user concurrency and reporting workloads expanded. Although an initial Amazon S3 data lake decoupled storage from compute, business users required standard SQL interfaces rather than Spark data pipelines. The team deployed Presto clusters on AWS EMR, switching their storage format from AVRO to Parquet to support ANSI SQL querying directly against S3. Utilizing a shared Hive metastore on Amazon RDS allowed Grab to adopt a shared-data multi-cluster architecture that isolated distinct workloads across dedicated compute clusters. This setup enabled rapid cluster scaling, streamlined failover, and matched Redshift performance on partitioned time-range queries.


### [How Grab Hires Engineers in Singapore](https://yomu.fyi/post/how-grab-hires-engineers-in-singapore.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Daniel Tay
- Published: Feb 16, 2017
- Type: Explainer

Grab maintains a selective engineering recruitment process in Singapore, with only three to five percent of candidates ultimately receiving an offer. The evaluation pipeline typically spans three to four weeks and includes an initial HR phone screen, an online coding round, and consecutive technical interviews. Recruiters assess candidate capability via public GitHub projects, presentations, and technical blogs rather than traditional CV qualifications alone. The company prioritizes experience in low-latency distributed systems and microservices design alongside strong cultural alignment with peer collaboration.


### [Round-robin in Distributed Systems](https://yomu.fyi/post/round-robin-in-distributed-systems.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Gao Chao
- Published: Sep 27, 2016
- Type: Explainer

Building client-side load balancing for Grab's Common Data Service prompted a move from AWS Elastic Load Balancers to DNS discovery due to persistent connection issues and unpredictable scaling events. After patching an open-source library that failed to rotate IP sequences properly, the author evaluated different Go patterns for round-robin routing. A mutex-protected array counter provides the simplest model for basic retrieval, though adding mutations requires careful lock coordination. Alternatively, a dedicated balancer goroutine receiving requests over nested channels enables explicit operation timeouts and centralized event handling at the cost of higher code complexity and channel creation overhead. The author recommends the mutex approach for resource fetching and the goroutine-based design for workload balancing.


### [Short Time Outs and Asynchronous Requests on Heroku](https://yomu.fyi/post/short-time-outs-and-asynchronous-requests-on-heroku.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Damien Mathieu
- Published: Dec 5, 2014
- Type: Explainer

Heroku’s short request timeout and H12 errors reflect a scaling problem: standard web servers process queued requests serially, so one long operation can block others. The post separates slow server-side actions, such as video processing or PDF generation, from file uploads, and recommends background processes for the former and direct browser uploads to Amazon S3 or Google Cloud Storage for the latter. When a client must wait for completion, it describes polling or a WebSocket, noting that WebSocket connections close after 55 seconds of inactivity and ordinary pages still need to respond within 30 seconds. Avoiding lengthy web-process work improves performance and scalability; rack-timeout for Ruby or expressjs’ timeout for Node can terminate requests after a few seconds, with a recommended limit no higher than 10 seconds.


[Newer posts](https://yomu.fyi/topic/scalability/page/18.md)