---
title: "Scalability"
description: "379 posts about Scalability, summarised, each linking to the original."
---

# Scalability
> 379 posts about Scalability, summarised, each linking to the original.

## Articles

### [Organizing 2000 Developers for BFCM in a Remote World](https://yomu.fyi/post/organizing-2000-developers-for-bfcm-in-a-remote-world.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 7, 2020
- Type: Problem & solution

Shopify organized a cross-company program to prepare its platform for Black Friday and Cyber Monday (BFCM), coordinating more than 2,500 people responsible for over 1,000 deploys across services. Engineering Program Managers absorbed administrative, communication, and reporting work so BFCM leads could focus on technical challenges, while departmental champions assigned preparation tasks. Preparation covered maintenance, game days, load tests, capacity increases, weekly tracking, and a Caution Tape bot requiring risk-to-value assessments for pull requests in critical areas. Deploys were slowed through smaller batches, longer canaries, and a planned weeklong lock while communication used Slack, GitHub, documentation, presentations, emails, and office hours. The effort coincided with $5.1+ billion in BFCM sales and load testing plus resource scaling paid off, but developer productivity declined amid inconsistent messaging; future plans called for a more balanced approach, automation, and reusable communication templates.


### [Capacity Planning at Scale](https://yomu.fyi/post/capacity-planning-at-scale.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 3, 2020
- Type: Explainer

Shopify’s BFCM preparation combines traffic forecasting, resource planning, and platform-wide scalability testing to reduce the risk of outages during its busiest sales period. The engineering team worked with data scientists to model expected traffic, reviewed historical numbers, added safety margins, and created a master Google Cloud resource plan covering CPUs, storage, machine types, locations, and project-level quotas. Because 2020 brought unusual uncertainty from COVID-19-driven changes in online commerce, Shopify modeled multiple demand scenarios and chose the more aggressive growth cases, while retaining buffers for changes and regional failover. It also ran the Oktoberfest scale-up on Google Kubernetes Engine, using internal load-testing tooling powered by go-lua and embedded Lua to simulate storefront browsing, cart additions, checkout, and mock payment processing. The exercise exposed bottlenecks, which teams fixed before repeating the test; the successful rerun increased confidence for BFCM and reinforced a longer-term aim of more automated, dynamic, and resilient systems.


### [Monitoro Built a Web Crawler Handling Millions of API Requests](https://yomu.fyi/post/monitoro-built-a-web-crawler-handling-millions-of-api-requests.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Published: Dec 2, 2020
- Type: Announcement

Supabase showcases numerous customer case studies illustrating how companies across various industries utilize its platform to build products, migrate infrastructure, and improve operational metrics. Several organizations report substantial cost and efficiency improvements, such as Shotgun achieving an 83% data infrastructure cost reduction and Good Tape saving 60% after moving to managed Postgres and authentication. In terms of developer velocity, Kayhan Space experienced an 8x boost in speed, while Quilia reduced development time by 75% through Data API and Row Level Security features. Other highlighted migrations include Mobbin moving 200,000 users from Firebase, Bree transitioning from Fauna, and Firecrawl switching from Pinecone to Supabase Vector for PostgreSQL vector embeddings.


### [TAYFA Built a No-Code Website Builder in Seven Days](https://yomu.fyi/post/tayfa-built-a-no-code-website-builder-in-seven-days.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Published: Dec 2, 2020
- Type: Explainer

Supabase compiles case studies detailing how organizations across industries build applications, scale infrastructure, and migrate databases. Organizations adopt the platform for capabilities including managed Postgres, Authentication, Row Level Security, Data APIs, and Supabase Vector for PostgreSQL vector embeddings. Reported operational outcomes include Shotgun reducing data infrastructure costs by 83 percent, Quilia cutting development time by 75 percent, and Good Tape achieving a 60 percent cost reduction. Several teams completed platform migrations to Supabase, including Mobbin migrating 200,000 users from Firebase, Bree moving from Fauna, and Firecrawl transitioning from Pinecone. Furthermore, organizations like Replenysh implemented one-time passwords within 24 hours, while Xendit shipped a production solution in under one week.


### [Xendit Built a Counter-Fraud Watchlist for the Fintech Industry](https://yomu.fyi/post/xendit-built-a-counter-fraud-watchlist-for-the-fintech-industry.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Published: Dec 2, 2020
- Type: Problem & solution

Payment processor Xendit needed a fast, cost-effective counter-fraud watchlist system to screen transactions against international sanctions lists in real time without relying on expensive providers like Worldcheck or Refinitiv. Leveraging their existing familiarity with Postgres, the team loaded parsed sanctions lists from the United Nations and the Indonesian government into Supabase. They implemented full-text search using the Postgres Trigram extension, enabling relevance scoring across queried records. A dedicated database function handles the search queries directly via Xendit's Python clients, allowing the entire solution to reach production in under one week. The deployment has operated reliably in production for nine months without issues, with potential future iterations planned around machine learning techniques.


### [Democratising Fare Storage at Scale Using Event Sourcing](https://yomu.fyi/post/democratising-fare-storage-at-scale-using-event-sourcing.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Sourabh Suman
- Published: Nov 23, 2020
- Type: Problem & solution

Grab's legacy system stored booking and fare details in a single relational table, creating a bloated booking entity that tracked only the latest fare state and hindered rapid feature iteration. To resolve scalability, stability, and debugging challenges across millions of daily bookings, the team developed Fare Storage using the Event Sourcing pattern. The new architecture persists all fare modification events chronologically in DynamoDB, backed by a cache for eventually consistent reads and message streaming for downstream processing. The platform employs optimistic locking with versioning to manage concurrent updates, enforces idempotency through client-generated transaction UUIDs, and delegates metadata serialization to an SDK to prevent storage API changes.


### [Supabase.js 1.0](https://yomu.fyi/post/supabase-js-1-0.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Paul Copplestone
- Published: Oct 30, 2020
- Type: Announcement

Supabase has launched Supabase.js 1.0, updating its client library ecosystem with breaking API changes, scalability enhancements, and new documentation tooling. The release introduces native TypeScript implementations across all underlying sub-libraries, including supabase-js, postgrest-js, gotrue-js, and realtime-js, which also enables auto-generated reference docs. In response to community feedback, database and authentication operations now return structured error objects alongside data payloads instead of throwing runtime exceptions. Realtime client scalability is improved by consolidating multiple active subscriptions into a single socket connection per Supabase client. Additionally, the release updates authentication workflows via the newly created gotrue-js library to support third-party logins and OAuth providers alongside revised method names.


### [Building Mental Models of Ideas That Don’t Change](https://yomu.fyi/post/building-mental-models-of-ideas-that-don-t-change.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 22, 2020
- Type: Explainer

Amid a constant stream of frameworks, languages, and platforms, the post argues that learning enduring principles is more useful than trying to absorb every new tool. It presents mental models as a checklist for prioritizing concepts, recording what matters, and making more informed decisions across engineering and management. Engineering examples include alerting on silent failures, queuing background jobs for faster responses and retries, using caching and denormalization for read-heavy workloads, and sharding or deliberate design choices for write-heavy systems. The post also recommends horizontal scaling for long-term growth, then turns to practices such as aligning incentives, clarifying product purpose, building rapport before feedback, and operating a few months ahead. The stated aim is to use these models to make more informed decisions rather than chase an ever-expanding list of new technologies.


### [New API Rate Limiter](https://yomu.fyi/post/new-api-rate-limiter.md)
- Company: [Zoom](https://yomu.fyi/company/zoom.md)
- Author: Benjamin Dean
- Published: Feb 14, 2020
- Type: Announcement

Zoom announced an updated V2 REST API Rate Limiter launching in March 2020 to replace its previous one-size-fits-all throttling system. Rather than applying uniform constraints across all subscription tiers, the new framework cross-references customer Account Types against API Request Groups categorized by infrastructure importance and load intensity. While most API rate limits are increasing, resource-intensive Dashboard APIs are reduced from 1 request per second to 6 or 12 requests per minute depending on account level. Throttled HTTP 429 responses will return X-RateLimit-Reset timestamps for per-minute limits and ISO8601 Retry-After timestamps for per-day limits. Additionally, the system introduces lock-key restrictions on specific user identifier path variables to restrict concurrency and enforce daily meeting creation caps.


### [Developing Zoom Marketplace Apps w/ ngrok](https://yomu.fyi/post/developing-zoom-marketplace-apps-w-ngrok.md)
- Company: [Zoom](https://yomu.fyi/company/zoom.md)
- Author: Tim Slagle
- Published: Feb 14, 2020
- Type: Problem & solution

Developers frequently use ngrok to establish fast, introspectable tunnels to localhost when building Zoom Marketplace applications instead of configuring complex reverse proxies like NGINX or Apache. However, using basic ngrok tunnels in production exposes systems to short URL expiration windows, single points of failure, and scalability bottlenecks. To maintain application availability, developers can purchase an ngrok license to secure a service-level agreement and support. Additionally, teams should reserve dedicated subdomains rather than using auto-generated endpoints, run ngrok as a monitored background service on a cloud provider or data center, and place a load balancer in front of the tunnel to handle heavy traffic.


### [How We Prevented App Performance Degradation from Sudden Ride Demand Spikes](https://yomu.fyi/post/how-we-prevented-app-performance-degradation-from-sudden-ride-demand-s.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Corey Scott
- Published: Jan 8, 2020
- Type: Problem & solution

Grab experienced severe system strain when sudden localized spikes in ride demand, triggered by events like heavy rain or concert dismissals, coincided with driver shortages. These localized bursts overloaded the platform and degraded the experience for users outside the affected areas. To mitigate this, engineers created the Spampede filter, a circuit-breaker mechanism placed at the start of the booking pipeline. The filter converts pickup locations into Geohash Integer buckets and partitions time using Unix timestamps, tracking unfulfilled requests in Redis with atomic increments and time-to-live expirations. When unallocated requests exceed configured thresholds within a specific bucket, the system immediately short-circuits new incoming bookings to protect overall platform stability.


### [Griffin, an Anti-fraud Risk Rule Engine Making Billions of Predictions Daily](https://yomu.fyi/post/griffin-an-anti-fraud-risk-rule-engine-making-billions-of-predictions.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Muqi Li
- Published: Oct 28, 2019
- Type: Problem & solution

Grab's Trust/Identity/Safety team built Griffin, an in-house anti-fraud risk rule engine designed to process billions of daily predictions across multiple business verticals. Initially, Grab managed fraud rules directly within backend service code, but escalating rule complexity, tight rule interdependencies, and translation gaps between data scientists and developers caused deployment delays and misfiring errors. To overcome the limitations and steep learning curves of third-party engines like Drools, the team separated the workflow into data orchestration and rule-based prediction. Griffin enables analysts and data scientists to author Python-based rules directly via a web portal and reload updated logic into memory without manual developer intervention. By eliminating I/O during rule evaluation and leveraging Gunicorn multi-processing, the engine handles over 100,000 queries per second at peak on six EC2 instances with single-prediction latencies under six milliseconds.


### [No More Forgetting to Input ERP Charges - Hello Automated ERP!](https://yomu.fyi/post/no-more-forgetting-to-input-erp-charges-hello-automated-erp.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Garvee Garg
- Published: Jul 31, 2019
- Type: Announcement

Grab launched an automated Electronic Road Pricing (ERP) fare calculation feature in Singapore to eliminate the need for driver-partners to manually track gantries and enter toll charges. Because Singapore gantries frequently adjust fares based on time and road conditions, manual entry often caused driver errors and revenue loss. Grab solved this by mapping precise geographical coordinates for every toll gate using satellite imagery and open data, matching frequent driver GPS pings against road layers and gantry locations. The engineering and operations teams also built an internal ERP Workflow tool to map ride trajectories and resolve driver dispute feedback within an average of one day. Following its rollout in Singapore, Grab began testing and planning regional expansion to Indonesia, Thailand, Malaysia, and the Philippines.


### [How We Built a Logging Stack at Grab](https://yomu.fyi/post/how-we-built-a-logging-stack-at-grab.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Daniel Kasen
- Published: Jul 31, 2019
- Type: Problem & solution

Grab needed a scalable logging platform to replace slow, fragmented systems that hindered debugging across their growing service fleet. Generating 25TB of daily logs, the team built a horizontally scalable Elasticsearch cluster configured via Ansible and monitored with Datadog. Although the initial proof of concept assigned all node roles (ingest, coordinator, master, and data) to every machine, operating at scale introduced major challenges with JVM heap exhaustion and cluster stability. The team resolved memory pressure and performance bottlenecks by tuning circuit breakers, lowering field data cache limits, adjusting shard allocations based on segment memory, and disabling translog compression during shard transfers.


### [Catwalk: Serving Machine Learning Models at Scale](https://yomu.fyi/post/catwalk-serving-machine-learning-models-at-scale.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Nutdanai Phansooksai
- Published: Jul 2, 2019
- Type: Problem & solution

As machine learning adoption expanded at Grab, individual teams created fragmented model serving solutions that duplicated engineering effort and required data scientists to handle underlying infrastructure. To resolve these inefficiencies, Grab developed Catwalk, a self-service machine learning model serving platform. The system runs TensorFlow Serving containers across a managed Kubernetes cluster integrated with Grab's observability stack. Data scientists deploy or update models simply by saving files using the tf.saved\_model API to dedicated Amazon S3 buckets, while Kubernetes automates orchestration, ingress routing, and pod autoscaling. Catwalk abstracts server management away from data scientists, shortens deployment timelines, and provides high availability during model version rollouts.


### [Designing Resilient Systems Beyond Retries (Part 2): Bulkheading, Load Balancing, and Fallbacks](https://yomu.fyi/post/designing-resilient-systems-beyond-retries-part-2-bulkheading-load-bal.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Michael Cartmell
- Published: Mar 25, 2019
- Type: Explainer

Software systems require mechanisms beyond retries to maintain resilience during downstream outages and high traffic. Bulkheading isolates failures across infrastructure, processes, thread pools, and connection limits, preventing a single failing component from degrading an entire system. Load balancing distributes traffic across backend pools via proxies, client-side libraries, lookaside services, or sidecars, often pairing with health checks to eliminate single points of failure. When operations fail unrecoverably, fallback strategies like silent failures, local defaults, stale cache reads, and dedicated backup services enable graceful degradation. Organizations like Grab implement these approaches using internal client-side load balancers backed by etcd, cache fallbacks in microservice frameworks, and redundant core backup services.


### [Designing Resilient Systems Beyond Retries (Part 1): Rate-Limiting](https://yomu.fyi/post/designing-resilient-systems-beyond-retries-part-1-rate-limiting.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Michael Cartmell
- Published: Mar 20, 2019
- Type: Explainer

Distributed systems that rely exclusively on retries and circuit breakers face severe failure risks, including retry storms and reliance on client-side configuration accuracy. Implementing server-side rate limiting serves as a critical defensive layer to safeguard services across evolving architectures. Throttling thresholds can be layered across per-client, per-endpoint, and server-wide granularities using algorithms such as leaky bucket or sliding windows. While local instance-level limits fail when downstream bottlenecks like databases saturate under horizontal scaling, global rate limiting coordinates traffic enforcement across entire service pools. Centralized rate limiters require asynchronous communication and fallback mechanisms to avoid becoming single points of failure or adding request path latency.


### [Recipe for Building a Widget: How We Helped to “Peak-Shift” Demand by Helping Passengers Understand Travel Trends](https://yomu.fyi/post/recipe-for-building-a-widget-how-we-helped-to-peak-shift-demand-by-hel.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Lara PuReum Yim
- Published: Mar 7, 2019
- Type: Problem & solution

Transport demand spikes during regular commuting hours often outpace driver availability, resulting in passenger wait times and fare surges. To mitigate these imbalances, Grab created the Travel Trends Widget for its mobile feed to redistribute ride requests toward off-peak windows. The widget uses machine learning forecasting to present historical supply-demand patterns alongside pricing trends for the upcoming two hours. To handle anticipated high query rates across millions of database entries, engineers periodically load precomputed trend data into an in-memory data structure rather than querying the database per request. The feature rolled out to feeds in Singapore and Jakarta within four weeks of initial development.


### [How We Simplified Our Data Ingestion & Transformation Process](https://yomu.fyi/post/how-we-simplified-our-data-ingestion-transformation-process.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Yichao Wang
- Published: Mar 3, 2019
- Type: Problem & solution

Grab evolved its real-time data ingestion pipeline after an initial architecture built on Spark Streaming and Python encountered operational complexity, node failures, and data loss from S3 eventual consistency. Because the streaming workload primarily handled event partitioning and ORC file generation, the team consolidated these tasks directly into an existing Golang processing service. They implemented sharded concurrent maps for high-throughput partitioning and optimized heap allocations to resolve memory bottlenecks. This refactor removed intermediate Avro conversions and intermediate storage hops. The simplified Go pipeline eliminated data loss and reduced processing lag from up to 13 minutes down to approximately 1 minute.


### [Understanding Supply & Demand in Ride-hailing Through the Lens of Data](https://yomu.fyi/post/understanding-supply-demand-in-ride-hailing-through-the-lens-of-data.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Aayush Garg
- Published: Feb 20, 2019
- Type: Explainer

Grab measures ride-hailing supply and demand across space and time to resolve geo-temporal allocation mismatches between moving drivers and ride-seeking passengers. The analytics pipeline defines supply as idle online drivers and demand as passengers checking fares within brief time slots, aggregating locations into geohashes. Each driver is mapped across neighbouring demand units and inversely weighted by straight-line distance, which yields the effective supply, supply-demand ratio, and supply-demand difference for each geographic polygon. Grab uses these aggregated metrics to identify marketplace imbalances, deploying driver heatmaps to shift excess supply and passenger travel trend widgets to defer time-insensitive ride requests.


[Newer posts](https://yomu.fyi/topic/scalability/page/17.md) · [Older posts](https://yomu.fyi/topic/scalability/page/19.md)