---
title: "Scalability"
description: "379 posts about Scalability, summarised, each linking to the original."
---

# Scalability
> 379 posts about Scalability, summarised, each linking to the original.

## Articles

### [Building hyperlocal GrabMaps](https://yomu.fyi/post/building-hyperlocal-grabmaps.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Adriana Lazar
- Published: Aug 30, 2023
- Type: Problem & solution

Southeast Asia presents unique mapping challenges, including fast-changing road restrictions, new roads appearing rapidly, and severe traffic congestion. To address these conditions, GrabMaps integrates hyperlocal data derived from local team knowledge, rides and deliveries GPS pings, and KartaView imagery with OpenStreetMap data. The pipeline downloads OpenStreetMap .pbf files for Asia and converts them into Parquet format using a Parquetizer. GrabMaps processes spatial data using Apache Spark and bi-directional graphs divided by geohash to detect complex features such as intersection links based on road parallelism, naming, direction, and shape geometry. Tagging these intersection links enables downstream navigation services like GrabNav to calculate shorter routing alternatives and provide more accurate pricing.


### [Streamlining Grab's Segmentation Platform with faster creation and lower latency](https://yomu.fyi/post/streamlining-grab-s-segmentation-platform-with-faster-creation-and-low.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Jake Ng
- Published: Aug 15, 2023
- Type: Problem & solution

Grab's Segmentation Platform previously stored user-to-segment mappings across individual rows in ScyllaDB, causing write bottlenecks during segment creation and read latencies too high for downstream consumers. To resolve these performance limitations, the team transitioned to storing segments as Roaring Bitmaps saved as single blobs in object storage. This compression strategy splits 32-bit integer user IDs into chunks across array, bitmap, and run containers based on data density, reducing a one-million-member segment to under one megabyte. An accompanying client SDK manages segment retrieval, decoding, update notifications, and least-recently-used in-memory caching. Consequently, consumers such as Grab's communications platform achieved peak throughput of 15,000 queries per second with sub-millisecond p99 read latencies.


### [What I learned taking Ramp Bill Pay from 0 to N](https://yomu.fyi/post/what-i-learned-taking-ramp-bill-pay-from-0-to-n.md)
- Company: [Ramp](https://yomu.fyi/company/ramp.md)
- Author: Pavel Asparouhov
- Published: Aug 1, 2023
- Type: Explainer

An engineer recounts lessons from taking Ramp’s Bill Pay from its first code in April 2021 to a publicly launched product that eventually became a product with multibillion annual total payment volume. For 0-to-1 work, the account emphasizes protecting core abstractions, including canonicalized payment statuses that reduce varied payment states to five, using mocked functions to coordinate parallel engineering, and reaching beta customers quickly. Customer feedback, session replay, and persistent iteration are presented as necessary because the first customer will usually expose substantial flaws. For 1-to-N growth, manual processes and duct tape must be prioritized for replacement according to user impact, error risk, and workload. Growing stakes also require testing, monitoring, contingency plans, gradual rollouts, QA, and documentation that helps engineers self-service onboarding.


### [Moving Fast by Moving Slow: How We Built Payments at Ramp](https://yomu.fyi/post/moving-fast-by-moving-slow-how-we-built-payments-at-ramp.md)
- Company: [Ramp](https://yomu.fyi/company/ramp.md)
- Author: Shreyan Jain
- Published: Jul 17, 2023
- Type: Problem & solution

Ramp describes why it delayed the launch of BillPay by about two months to replace its existing, ad hoc bank-transfer infrastructure with a new payments service. The earlier system encoded product-specific concepts such as employee bank accounts and reimbursement transfers, forcing each new use case to add bespoke data models, jobs, and scheduling logic. This friction became especially costly as the team anticipated additional payment products and vendor payments scheduled for future dates. The redesign used generic primitives—ACHDetails and TransferCanonical—to separate business logic from money-movement logic; it also treats failed transfers and retries as concerns requiring an auditable identity, history, and consistent provider mapping. The post argues that stress-testing assumptions, modeling relevant degrees of freedom, limiting conditional logic, and building platforms after recurring patterns can make future products faster to launch.


### [pgvector 0.4.0 performance](https://yomu.fyi/post/pgvector-0-4-0-performance.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: egor\_romanov,pavel
- Published: Jul 13, 2023
- Type: Benchmark

The post benchmarks pgvector 0.4.0 for approximate nearest-neighbor search and assesses whether PostgreSQL can support production-scale vector workloads. Using the ANN Benchmarks methodology, pre-warmed RAM, the dbpedia-entities-openai-1M dataset of 1 million 1,536-dimensional OpenAI embeddings, and Supabase ARM instances, it varies probes, lists, concurrency, and compute size. On an 8-core, 32 GB instance, pgvector reached 0.91 accuracy@10 at 380 QPS with 10 probes and 0.98 accuracy at 140 QPS with 40 probes; a 64-core, 256 GB server reached about 1,800 QPS at 0.91 accuracy. The results indicate predictable scaling, while higher lists can improve query speed at the cost of index-build time, and the roadmap identifies HNSW, product quantization, and parallel index builds as planned improvements.


### [Message Center - Redesigning the messaging experience on the Grab superapp](https://yomu.fyi/post/message-center-redesigning-the-messaging-experience-on-the-grab-supera.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Jonathan Lee
- Published: Apr 17, 2023
- Type: Problem & solution

Grab redesigned its messaging infrastructure from GrabChat to Message Center to overcome two-party chat limitations and support complex superapp requirements like group conversations and varied user roles. The architecture separates core processing logic from message delivery by splitting the system into a backend processor and an independently scalable postman service. Communication relies on an in-house TCP gateway named Hermes that proxies client payloads via gRPC, alongside Apache Kafka streams and Amazon SQS delay queues. Custom client-server acknowledgements and a DynamoDB event store ensure reliable message delivery even across dropped mobile TCP connections and offline reconnects.


### [Supabase Storage v3: Resumable Uploads with support for 50GB files](https://yomu.fyi/post/supabase-storage-v3-resumable-uploads-with-support-for-50gb-files.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: fabrizio,inian
- Published: Apr 12, 2023
- Type: Announcement

Storage v3 adds resumable uploads, quality and format filters, a Next.js loader, WebP rendering, presigned upload URLs, and per-bucket size and file-type limits. Resumable uploads divide files into chunks, emit progress events, and allow interrupted transfers to continue; Pro Plan and higher users can upload files up to 50GB, up from 5GB. Rather than expose S3’s proprietary protocol, the implementation uses the open TUS protocol over HTTP through the Node.js Storage-API service and tus-node-server, with Postgres RLS integration. To support multiple TUS instances behind a load balancer, transactional Postgres Advisory locks serialize access to each S3 object and help address concurrent uploads, while the release rolls out gradually over the next month.


### [What Being a Staff Developer Means at Shopify](https://yomu.fyi/post/what-being-a-staff-developer-means-at-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Mar 29, 2023
- Type: Explainer

The post defines a staff developer at Shopify as an individual contributor whose scope and seniority can match a manager’s, while emphasizing that the role is a form of leadership rather than simply a more senior senior developer. Technical excellence comes from varied, challenging project experience, including large-scale systems, refactors, technology shifts, mistakes, and learning to choose appropriate solutions under constraints. The role also pairs with a team manager, translating product and business objectives into technical proposals, judging scope and infrastructure needs, finding recurring product problems, and saying no when effort does not match return. Its impact is measured partly through enabling others via reviews, mentorship, sponsorship, and support, leading to an adaptable leader comfortable with ambiguity whose responsibilities evolve with company needs.


### [Unlocking Real-time Predictions with Shopify's Machine Learning Platform](https://yomu.fyi/post/unlocking-real-time-predictions-with-shopify-s-machine-learning-platfo.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Mar 10, 2023
- Type: Explainer

Shopify’s Merlin machine learning platform was expanded to support online inference, allowing trained models to return predictions as input arrives rather than through periodic batch jobs. The design targets varied Shopify use cases and requirements, including support for TensorFlow, PyTorch and XGBoost, low latency, autoscaling, rolling deployments, observability and model updates. Each use case runs as a dedicated service on Google Kubernetes Engine, with a Python serving layer that loads models and artifacts, exposes an inference endpoint, and can use the Comet ML model registry and Pano feature store. Separate staging and production configurations let teams tune resources and test model versions before release; Shopify reports that online inference is providing the intended low latency, scalability and faster iteration, while ensemble inference, platform-based monitoring and continuous training remain planned areas.


### [HappyTeams unlocks better performance and reduces cost with Supabase](https://yomu.fyi/post/happyteams-unlocks-better-performance-and-reduces-cost-with-supabase.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Published: Feb 16, 2023
- Type: Problem & solution

HappyTeams, a bootstrapped startup, needed its database to support thousands of users and continued feature development without consuming scarce technical resources. Its Heroku Postgres deployment had developed scaling problems, including sluggish or uneven performance and limits on concurrent connections, prompting a database change during an application rebuild in early 2022. The company migrated to Supabase in less than an hour using pg\_dump and pg\_restore, and adopted PgBouncer for connection pooling; it selected the service for performance, connection flexibility, support and documentation, onboarding, and lower costs. The source reports that Supabase removed the performance issue, made pooling effortless, and helped HappyTeams release features faster, including a comments feature completed in two days, while the company considered expanding into Edge Functions and the API client.


### [Performance Testing At Scale—for BFCM and Beyond](https://yomu.fyi/post/performance-testing-at-scale-for-bfcm-and-beyond.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jan 27, 2023
- Type: Benchmark

Shopify treats Black Friday Cyber Monday (BFCM) as its most critical period, and uses full-scale BFCM Scale Testing to assess whether the platform can handle peak traffic. Preparation combines capacity planning, resiliency testing, single-application performance testing, and repeated exercises that scale core services to their BFCM profiles before generating load. The hybrid architecture centers on Shopify Core, a monolithic Ruby application supported by services for storefront rendering, analytics, fraud analysis, payments, webhooks, and more. Five load flows cover browsing and buying, admin activity, flash sales, Storefront APIs, and Hydrogen and Oxygen, with Cronograma layering requests gradually and providing an abort switch. The exercises surfaced connection-count pressures, a Kubernetes ingress-nginx endpoint limit, and uneven load-balancer distribution; mitigations included refactoring, scaling dependent services, increasing pod capacity, and dedicated nginx node pools.


### [3 (More) Tips for Optimizing Apache Flink Applications](https://yomu.fyi/post/3-more-tips-for-optimizing-apache-flink-applications.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 13, 2022
- Type: Tutorial

Shopify presents three additional practices for optimizing large stateful Apache Flink applications, focusing on parallelism, sink capacity, and combining heterogeneous sources. It recommends starting with execution-environment parallelism, then matching task-manager capacity to the highest parallelism value; for example, parallelism 100 requires 25 task managers with four slots each. Sink bottlenecks can propagate backpressure upstream, so batching writes and correcting data skew—including with key bucketing when a low-cardinality or uneven key is unavoidable—can improve distribution, though bucketed results must later be combined. For archived Kafka topics, HybridSource exposes cloud-storage history and the live Kafka topic as one ordered logical DataStream, automatically switching after the archive is exhausted and using object-storage partitions to accelerate historical backfills.


### [Planning in Bets: Risk Mitigation at Scale](https://yomu.fyi/post/planning-in-bets-risk-mitigation-at-scale.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 6, 2022
- Type: Explainer

Preparing for BFCM and other high-traffic events, Shopify describes risk mitigation for a globally distributed, complex, interconnected infrastructure platform that peaked at 75.98M requests per minute, or 1.27M per second. At this scale, identifying and mitigating every possible failure is impossible, so the process starts with “what could go wrong” exercises, then uses voting, expert review of likelihood and severity, and assigned ownership to prioritize action. Decision makers are empowered to weigh risks and rewards, while summarized findings, including key risks and single points of failure, are shared to maintain alignment and awareness. The post frames engineering choices as probabilistic bets: decision quality should be judged separately from outcomes, and uncertainty cannot be eliminated, only addressed through thoughtful prioritization and communication.


### [Using Server Sent Events to Simplify Real-time Streaming at Scale](https://yomu.fyi/post/using-server-sent-events-to-simplify-real-time-streaming-at-scale.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 30, 2022
- Type: Problem & solution

Shopify’s 2022 BFCM Live Map needed to deliver real-time sales and product data to thousands of concurrent users while processing data from millions of merchants. Its 2021 architecture used WebSocket delivery to a presentation layer, which stored messages in a mailbox that clients polled at least every 10 seconds, creating latency and bottlenecks. Shopify replaced that path with a streamlined Flink-based pipeline and a Golang SSE server that subscribes to Kafka topics and pushes JSON data to registered, authenticated clients as soon as it is available. The SSE server ran horizontally behind Nginx load balancers, with a configurable Java client used to load-test concurrent connections. The 2022 system maintained 100 percent uptime, delivered SSE data within milliseconds of availability, and visualized data within 21 seconds of creation, including pipeline processing.


### [Graph for fraud detection](https://yomu.fyi/post/graph-for-fraud-detection.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Min Chen
- Published: Nov 24, 2022
- Type: Problem & solution

Rapid expansion across multiple business verticals introduced diverse, evolving fraud patterns at Grab, making traditional rule engines and decision trees inadequate due to their heavy reliance on extensive feature engineering and labeled data. To address cold starts and detect unknown fraud, the engineering team implemented a semi-supervised Relational Graph Convolutional Network (RGCN) across millions of connected ecosystem entities. The model propagates information along neighborhood nodes through graph convolutional layers to produce node embeddings and output fraud probabilities. Trained on graphs where only a small percentage of nodes are labeled, the RGCN achieved an AUROC close to 1 and distinctly separated fraudulent embeddings from genuine ones. The approach also offers high explainability by surfacing dense clusters of shared physical devices and addresses ongoing challenges in real-time prediction and noisy connections.


### [Implementing Server-Driven UI Architecture on the Shop App](https://yomu.fyi/post/implementing-server-driven-ui-architecture-on-the-shop-app.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 22, 2022
- Type: Explainer

The Shop Store team implemented a server-driven UI architecture for the Shop App’s Store Screen to personalize sections and layouts for different merchants. Previously, the client rendered a fixed layout, limiting customization, experiment timing, and the speed of fixes after weekly app releases. The new design uses template processing, orchestration, data loading, and a shop-server GraphQL layer, with ProductsSection and CollectionsSection types and configurable GridLayout or ShelfLayout options. On the client, ServerDrivenStoreScreen and StoreSectionContainer render the sections returned by the server, while default layouts allow older app versions to handle unfamiliar section or layout types. The architecture enables backend-controlled experiments and section changes without requiring an app release, while giving the product team a system intended to grow with the store experience.


### [Reducing BigQuery Costs: How We Fixed A $1 Million Query](https://yomu.fyi/post/reducing-bigquery-costs-how-we-fixed-a-1-million-query.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 3, 2022
- Type: Problem & solution

During the infrastructure work for a marketing tool, Shopify's team found a BigQuery query that would have processed about 75,462,743,846 bytes per request and cost nearly $1 million monthly at an estimated 60 requests per minute. The query served a pipeline ingesting one billion rows through Apache Flink, with state managed by RocksDB and streaming requests from Apache Kafka; scaling beyond the release made ingestion unsustainable. To support general availability, the team evaluated an external SQL warehouse that could load Parquet atomically, handle 60 requests per minute, and export results to Google Cloud Storage, then clustered a dataset on two feature columns used in WHERE clauses. Running the same query on the clustered table reduced billed data to 508.1 MB and identified 108.3 MB scanned, lowering estimated monthly cost to about $1,370.67; the post also recommends selecting needed columns, partitioning tables, and using free previews instead of exploratory queries.


### [Using mobile sensor data to encourage safer driving](https://yomu.fyi/post/using-mobile-sensor-data-to-encourage-safer-driving.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Laiyi Lin
- Published: Oct 25, 2022
- Type: Problem & solution

Dedicated vehicular inertial measurement units are prohibitively expensive to install across large fleets, while customer feedback lacks the coverage required to assess driving safety comprehensively. To monitor driver behavior at scale, Grab processes smartphone sensor data collected via a mobile telematics SDK at high sampling frequencies. A custom orientation algorithm calculates Euler roll and pitch angles from gravity and resolves yaw through principal component analysis to align phone measurements with vehicle axes. The pipeline applies low-pass or Kalman filtering to handle varying noise levels, followed by threshold peak-finding algorithms to detect harsh acceleration, braking, and cornering. Speeding events are identified by comparing GPS speeds against known map limits or aggregated speed profiles derived from free-flowing traffic.


### [Leveraging Go Worker Pools to Scale Server-side Data Sharing](https://yomu.fyi/post/leveraging-go-worker-pools-to-scale-server-side-data-sharing.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Sep 9, 2022
- Type: Problem & solution

Shopify’s Server Pixels service validates, processes, augments, and produces more than one billion customer events daily, but rising traffic caused Kafka input-topic consumption lag when downstream processing slowed. Its original design could spawn an unlimited number of goroutines while processing or producing event batches, consuming CPU and memory and threatening the five-minute event-processing SLO. The service was redesigned around fixed-size Go worker pools: workers receive Job values through channels, process them via reusable work functions, and use wait groups and context cancellation for orderly shutdown. Load testing showed 15 workers in both processor and producer raised throughput to 12.9 thousand events per second per pod, while 50 workers reached 19.3 thousand; subsequent improvements reached 21 thousand, a 170% increase over 7.75 thousand, and the service handled a maximum of 46 thousand events per second during BFCM 2021.


### [Realtime: Multiplayer Edition](https://yomu.fyi/post/realtime-multiplayer-edition.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: wenbo,stas
- Published: Aug 18, 2022
- Type: Announcement

Supabase announces the general availability of Realtime’s multiplayer features, Broadcast and Presence, alongside a new channel-based architecture and supabase-js support. Realtime originally used Postgres logical replication, rather than LISTEN/NOTIFY, to avoid the latter’s 8,000-byte payload limit and the latency of querying each change from the database. In the new design, Channels act as rooms that support ephemeral publisher/subscriber Broadcast messages, shared Presence state, and Postgres change events; Presence uses Phoenix Tracker, a delta-based CRDT, for eventually consistent synchronization and removes disconnected users automatically. Realtime now runs as a distributed Elixir and Phoenix cluster on Fly, with clients connecting to the nearest node and nodes communicating across a backbone network to improve connection speed, stability, and latency. Free-plan active projects and projects created from August 18 have access, while other projects are scheduled for migration over the following weeks.


[Newer posts](https://yomu.fyi/topic/scalability/page/13.md) · [Older posts](https://yomu.fyi/topic/scalability/page/15.md)