---
title: "Scalability"
description: "379 posts about Scalability, summarised, each linking to the original."
---

# Scalability
> 379 posts about Scalability, summarised, each linking to the original.

## Articles

### [Supporting large campaigns at scale](https://yomu.fyi/post/supporting-large-campaigns-at-scale.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Jie Zhang
- Published: Apr 1, 2022
- Type: Problem & solution

Grab developed a batch job service within its Trident automation engine to execute multi-step marketing campaigns for millions of users simultaneously. The system replaces sequential, single-server execution with a distributed architecture powered by Apache Kafka, which distributes batches of 100 users across server clusters using hashed partition keys. To reduce network overhead and queries per second, downstream reward and messaging services introduced batch endpoints backed by bulk database queries, decreasing API latency by up to 85%. Grab further optimized performance by sharding Kafka topics by country and action type to prevent long-running reward tasks from blocking time-sensitive messaging workloads. Additionally, making terminal messaging calls asynchronous allows subsequent batch processing to proceed without waiting for message delivery confirmations.


### [Introducing Supabase Enterprise](https://yomu.fyi/post/introducing-supabase-enterprise.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Rory Wilding, Paul Copplestone
- Published: Mar 30, 2022
- Type: Announcement

Supabase announced the launch of its Enterprise Plan alongside new features integrated across existing tiers. The Pro Plan now merges pay-as-you-go billing with a default twenty-five dollar monthly spend cap, and users gain access to self-serve database add-ons for on-demand scaling. A new Log Explorer powered by the Logflare acquisition allows querying logs directly with SQL, providing ninety days of retention on the Enterprise tier. Dedicated enterprise capabilities include Point-in-Time Recovery powered by the open-source tool WAL-G, custom Prometheus scraping endpoints for observability, and guaranteed support service level agreements. Additionally, Supabase achieved SOC2 Type 1 compliance and introduced Elixir Livebooks to monitor PgBouncer connection pools.


### [Abacus - Issuing points for multiple sources](https://yomu.fyi/post/abacus-issuing-points-for-multiple-sources.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Chandrakanth
- Published: Mar 1, 2022
- Type: Problem & solution

Grab needed a centralised points management architecture to issue loyalty points across a growing catalog of products, membership tiers, and external partner exchanges. To address this, the engineering team built Abacus, an issuance platform designed to process millions of daily transactions with high availability. The system ingests completed transaction streams or API calls, dynamically computes points via configured multipliers, and passes calculations through Amazon Simple Queue Service queues. Once the Point Awarding module updates a persistent ledger, Abacus notifies consumers, emits events to Kafka for downstream consumers, and recalculates rolling point expiration dates.


### [Exposing a Kafka Cluster via a VPC Endpoint Service](https://yomu.fyi/post/exposing-a-kafka-cluster-via-a-vpc-endpoint-service.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Fabrice Harbulot
- Published: Feb 18, 2022
- Type: Problem & solution

To replace VPC peering and reduce attack surfaces, Grab exposed a multi-Availability Zone Apache Kafka cluster in its main AWS VPC to clients in a separate GrabKios VPC using AWS VPC Endpoint Service. Because Kafka requires clients to establish deterministic connections to individual brokers, the team configured a Network Load Balancer with unique TCP ports and dedicated target groups for each broker alongside a shared bootstrap port. They added custom listeners on the Kafka brokers to advertise endpoints using private Route 53 CNAMEs rather than raw interface hostnames. To eliminate unnecessary cross-AZ network latency and data transfer costs, the architecture was refined to advertise AZ-specific private CNAMEs mapped directly to zonal endpoint interfaces.


### [How Grab built a scalable, high-performance ad server](https://yomu.fyi/post/how-grab-built-a-scalable-high-performance-ad-server.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Anthony McCallum
- Published: Feb 11, 2022
- Type: Problem & solution

Grab transitioned from an off-the-shelf MVP to an in-house ad serving system to accommodate business scale, hyperlocal requirements, and machine learning personalization. The architecture orchestrates core microservices and data pipelines across sequential steps: targeting, capping, pacing, scoring, ranking, pricing, and tracking. ElasticSearch serves as the targeting ads repository, while ScyllaDB acts as the high-throughput stats store fed by Kafka streams and data pipelines. The system operates on key engineering principles including parallelization and tuned latency limits, graceful fallbacks for slow dependency calls, and a unified server serving all ad types across the superapp.


### [Shopify's Playbook for Scaling Machine Learning](https://yomu.fyi/post/shopify-s-playbook-for-scaling-machine-learning.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Feb 11, 2022
- Type: Explainer

Shopify describes how its machine-learning practice grew from a first order-fraud product into a portfolio used across products such as Shopify Capital, product categorization, and Help Center search. The playbook starts with choosing a user-important problem, validating trustworthy and accessible data, tracing downstream dependencies, understanding existing solutions, and optimizing for product outcomes rather than model scores alone. To productionize models, Shopify recommends well-defined training, verification, and testing pipelines, deployment decisions based on volume and user commitment, and platform practices that encode validated standards without abstracting too early. Its fraud pipeline builds models in Python, serializes them with PMML for deployment in Ruby-based production systems, and uses Apache Airflow to schedule operations. The account concludes that scaling is a full-stack problem requiring data scientists and data engineers to align on one roadmap and goal.


### [Cloud, Load, and Modular Code: What 2022 Looks Like for Shopify](https://yomu.fyi/post/cloud-load-and-modular-code-what-2022-looks-like-for-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jan 20, 2022
- Type: Announcement

Shopify’s 2021 BFCM four-day event exposed the scale its commerce platform must absorb: peak app-server traffic exceeded 32 million requests per minute, load balancers surpassed 34 million RPM, and a flash sale used more than 20% of total computing capacity. The post frames larger, spikier sales, mobile and social commerce, and expanding buying environments as enduring engineering challenges for supporting millions of merchants. Planned work includes evolving the large Rails codebase into modular components, moving storefront rendering out of the monolith, expanding Spin cloud development environments, and making in-house load testing part of product development. The storefront split and refactoring produced four-times-faster cache fill rates and five-times-faster page rendering, while YJIT ran 31% faster than interpreted CRuby and 26% faster than MJIT; Shopify also remains committed to Rails and React Native.


### [Search at Shopify—Range in Data and Engineering is the Future](https://yomu.fyi/post/search-at-shopify-range-in-data-and-engineering-is-the-future.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jan 14, 2022
- Type: Explainer

Search at Shopify is presented as a case for range across data science and engineering, rather than narrow specialization in either discipline. The article describes a team that treats work as “search” instead of separating data and engineering responsibilities: engineers learn experimentation and model evaluation, while data scientists build performant, testable, maintainable production systems. It frames search as a sequence of trade-offs among relevance, correctness, scalability, performance, stability, and maintainability, arguing that people who understand only half the constraints can produce siloed handoffs, deployment delays, or suboptimal systems. Shopify’s stated model is a single collaborative team with shared planning and execution, intended to reduce territorial boundaries and support smart, fast, scalable search. The article extends this principle to personalized and conversational products, concluding that future work will require practitioners to move between data and engineering.


### [Scaling Shopify's BFCM Live Map: An Apache Flink Redesign](https://yomu.fyi/post/scaling-shopify-s-bfcm-live-map-an-apache-flink-redesign.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 10, 2021
- Type: Problem & solution

Shopify’s 2021 BFCM live map had to scale from more than 1.7 million merchants while adding product trends and unique shoppers, after the prior Cricket-and-Redis backend showed volume and delivery weaknesses. The redesign placed Apache Flink on the critical path to filter irrelevant checkout events, reducing Cricket’s workload to about one percent of event volume, while cross-region sharding and active-active deployment supported availability. Flink computed new metrics, including 500 product categories ranked by sales-volume change every five minutes, while Cricket retained existing metric computation and deduplication. In load tests, the pipeline handled 1 million checkout events per second, and its jobs ran at 100 percent uptime during BFCM without backpressure or manual intervention. Redis still caused high CPU and memory pressure, producing brief arc-visual errors for a small sample of users, so Shopify plans a simpler Flink-centered design.


### [Upgrading MySQL at Shopify](https://yomu.fyi/post/upgrading-mysql-at-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 2, 2021
- Type: Problem & solution

In September 2021, Shopify completed its cutover from Percona Server 5.7.21 to 5.7.32 across a MySQL fleet spanning hundreds of horizontally sharded databases and thousands of virtual machines. The upgrade targeted replication improvements from transaction writesets, along with security and stability fixes, while avoiding the larger risks of a major-version change. The Database Platform team extended Mason into a declarative topology system and used Chef roles to provision upgraded VMs, restore backups, run mysql\_upgrade when required, configure replication, and populate replacement topologies. Extensive rollback testing exposed a critical downgrade bug; after patching 5.7.21, Shopify proceeded in staggered batches beginning August 25 and completed the final shards on September 7, while documenting a repeatable process intended to make future upgrades faster and safer.


### [Using real-world patterns to improve matching in theory and practice](https://yomu.fyi/post/using-real-world-patterns-to-improve-matching-in-theory-and-practice.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Tenindra Abeywickrama
- Published: Nov 22, 2021
- Type: Problem & solution

Continuous ride-hailing assignment relies on solving the minimum weight bipartite matching problem between passengers and driver-partners. While traditional implementations assume a precalculated cost matrix, computing shortest-path travel times across large road networks dominates total execution time. Researchers introduced an Incremental Kuhn-Munkres algorithm that leverages the spatial locality of optimal matches to compute edge costs on demand. The approach integrates priority queues and lower-bounding techniques with refinement rules to avoid evaluating distant pairs while guaranteeing the same optimal assignment. Evaluated on Singapore road network data and real Grab production workloads, the incremental techniques reduced exact cost calculations and decreased assignment running times by over an order of magnitude.


### [Supabase $30m Series A](https://yomu.fyi/post/supabase-30m-series-a.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Paul Copplestone
- Published: Oct 28, 2021
- Type: Announcement

Supabase announced a $30 million Series A funding round led by Coatue, bringing its total funding to $36 million. The company aims to provide an open source alternative to Firebase while addressing the scalability limitations typical of abstracted database platforms. Rather than building a compatibility layer, Supabase provisions a full PostgreSQL database for every project alongside integrated authentication, storage, RESTful APIs, and realtime data streaming. Platform features rely directly on core PostgreSQL primitives, including Row Level Security policies, database triggers, and logical replication streams. Funding will support long-term development of a cloud-native PostgreSQL platform offering database branching, distributed read-write nodes, scalable storage, and ephemeral compute that scales down to zero.


### [Winning AI4TSP: Solving the Travelling Salesperson Problem with Self-programming Machines](https://yomu.fyi/post/winning-ai4tsp-solving-the-travelling-salesperson-problem-with-self-pr.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 6, 2021
- Type: Benchmark

The 2021 AI4TSP Competition addressed the Travelling Salesperson Problem when travel times are sampled from probability distributions and locations carry rewards and time windows. With n locations, the number of possible tours is n!, making exhaustive enumeration impractical, while the competition also required decisions under uncertain travel outcomes. One track required a fixed tour optimized for expected performance across scenarios; the other allowed an adaptive policy to select the next location as time elapsed, with results evaluated across many instances and realizations. The Shopify-led team used dialectic search for the fixed-tour track and deep reinforcement learning for the on-the-fly track, and won first prize in both.


### [Shard Balancing: Moving Shops Confidently with Zero-Downtime at Terabyte-scale](https://yomu.fyi/post/shard-balancing-moving-shops-confidently-with-zero-downtime-at-terabyt.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Sep 24, 2021
- Type: Explainer

Shopify’s MySQL shards can become unevenly utilized as merchants grow after being assigned to arbitrary shards, increasing over-utilization risk while leaving other capacity underused. The post describes a strategy for selecting shop placements using historical utilization and traffic data, simulations, and potentially other metrics such as shop size, GMV, and move time. It then explains online shop migration with Ghostferry, which batch-copies records and tails MySQL’s binary log before a brief cutover updates the routing table and directs new writes to the target pod. Verification checks data completeness, integrity, and stale traffic before old data is pruned; the migration system’s central algorithm is also represented in a TLA+ formal specification. The stated outcome is a more balanced infrastructure that can isolate high-traffic merchants across pods with virtually zero consumer-facing downtime.


### [Making Shopify’s Flagship App 20% Faster in 6 Weeks Using a Novel Caching Solution](https://yomu.fyi/post/making-shopify-s-flagship-app-20-faster-in-6-weeks-using-a-novel-cachi.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Sep 22, 2021
- Type: Problem & solution

Shop’s home feed, the app’s most used feature, aggregated orders from millions of Shopify and non-Shopify merchants plus tracking data from dozens of carriers, accounted for 30% of database load and affected application performance. Before each database update, the custom write-through cache marks a user’s cache invalid, applies the write, then refreshes and revalidates it. To handle concurrent updates, it uses a separate pending-writes key and a short-expiry key to distinguish active writes from interrupted processes, while Active Record Concerns preserve the existing ORM interface. After a staged validation rollout, the cache reduced database load by 15%, overall app latency by about 20%, and overall GraphQL latency by 20%, with a double-digit decrease in CPU usage.


### [Using Rich Image and Text Data to Categorize Products at Scale](https://yomu.fyi/post/using-rich-image-and-text-data-to-categorize-products-at-scale.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Sep 8, 2021
- Type: Explainer

Shopify revisited product categorization as its platform expanded to millions of merchants selling billions of products across diverse industries. The model uses product titles, descriptions, types, vendors, collections, tags, and images, converting them into embeddings with Multilingual BERT and MobileNet-V2 before hierarchical classification across the seven-level Google Product Taxonomy. Each taxonomy level is treated as a separate multi-class task, with parent predictions fed into child predictions and weighted losses combined during training; confidence thresholds filter predictions at each level. The upgraded system increased leaf precision by 8% while almost doubling coverage, although thresholds require balancing accuracy, coverage, merchant experience, and performance in sensitive categories, and future work includes addressing data imbalance and adding merchant-level features.


### [Supabase Reports and Metrics](https://yomu.fyi/post/supabase-reports-and-metrics.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Div Arora
- Published: Jul 29, 2021
- Type: Announcement

Supabase introduced a dedicated Reports section and updated project home pages to deliver observability across hosted Postgres projects and complementary services. Because developers can run unconstrained database queries and custom workloads, monitoring both project-level service usage and underlying infrastructure metrics is essential. The new dashboard displays one month of historical data, including CPU and memory usage alongside request volumes for Database, Auth, and Storage APIs. To collect metrics across hundreds of thousands of targets without high latency, Supabase implemented a federated monitoring architecture running regional Prometheus instances that scrape local exporters and aggregate subsets into a central VictoriaMetrics instance. All project databases now also launch with the pg\_stat\_statements extension enabled for future query statistics.


### [Mobbin uses Supabase to authenticate 200,000 users](https://yomu.fyi/post/mobbin-uses-supabase-to-authenticate-200-000-users.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Published: Jul 28, 2021
- Type: Problem & solution

Mobbin originally launched its mobile design search platform using Firebase, quickly growing to hundreds of thousands of registered users. As the platform scaled, the team faced user friction because Firebase Authentication did not merge accounts sharing the same email address across different providers. Furthermore, mutative Firebase Function failures threatened document data integrity, and multi-round-trip document queries drove high monthly costs across five million API requests. To resolve these challenges, Mobbin migrated its backend to Supabase, adopting its GoTrue authentication service and underlying Postgres relational database. This migration resolved authentication conflicts, safeguarded data consistency, and reduced monthly infrastructure expenses since Supabase does not charge on a per-API-request basis.


### [A Guide to Running an Engineering Program](https://yomu.fyi/post/a-guide-to-running-an-engineering-program.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 21, 2021
- Type: Explainer

Shopify describes a playbook for running large engineering programs as platform complexity grows across products, tooling, and architecture. The framework begins with a Program Plan covering the problem statement, objectives, guiding principles, definition of done, risks, mitigation, staffing, scope, and a path to completion. Execution uses six-week cycles that revisit unfinished goals, regressions, emerging risks, and the planned route to completion; the described program spans 200 people, nine sub-organizations, and 92 projects. Supporting rituals include weekly updates, status checks, risk and escalation triage, cycle reviews, retrospectives, RFCs, and performance testing with Lua scripts orchestrated by Genghis. The post concludes that engineering program management must adapt to each organization and that a combination of scheduled and ad hoc rituals helped Shopify pursue its goals.


### [Deep Learning over the Internet: Training Language Models Collaboratively](https://yomu.fyi/post/deep-learning-over-the-internet-training-language-models-collaborative.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Max Ryabinin, Lucile Saulnier
- Published: Jul 15, 2021
- Type: Problem & solution

Pretraining state-of-the-art Transformer language models typically requires vast computing resources and expensive hardware clusters that are inaccessible to many researchers. Standard distributed deep learning struggles over public Internet infrastructure because consumer connections suffer from limited bandwidth, dynamic peer disconnections, and restrictive firewalls. To resolve these networking bottlenecks, Distributed Deep Learning in Open Collaborations (DeDLOC) introduces an adaptive training framework that accumulates massive batch sizes across heterogeneous volunteer devices before applying optimizer updates. The framework splits gradient vectors dynamically based on participant connection speeds and allows firewalled nodes to contribute data without performing aggregation. In a practical deployment with 40 volunteers, the 18-million-parameter sahajBERT model achieved Bengali downstream benchmark performance comparable to much larger multilingual models trained on dedicated supercomputers.


[Newer posts](https://yomu.fyi/topic/scalability/page/15.md) · [Older posts](https://yomu.fyi/topic/scalability/page/17.md)