---
title: "Scalability"
description: "379 posts about Scalability, summarised, each linking to the original."
---

# Scalability
> 379 posts about Scalability, summarised, each linking to the original.

## Articles

### [Supabase Series B](https://yomu.fyi/post/supabase-series-b.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Paul Copplestone
- Published: Aug 12, 2022
- Type: Announcement

Supabase announced an $80 million Series B in May, bringing total funding to $116 million, and reported growth to more than 150,000 hosted PostgreSQL databases and 110,000 developers. Community metrics also increased, with Discord membership rising from 4,000 to 8,000, Twitter followers from 18,000 to 37,000, and GitHub stars from 19,000 to 36,000. The round is led by Felicis, with Coatue, Lightspeed, and Square Peg Capital joining, while Supabase describes its open-source support through contributor payments, maintainer hiring, sponsorship, and community collaboration. The company is moving into its third phase, focused on cloud-native PostgreSQL capabilities including branching, scalable storage, distribution, ephemeral compute, snapshots, and time travel. It invested $500,000 in OrioleDB and hired a developer to work on it, while stating that OrioleDB is not currently used on the Supabase platform and that the company will not run a PostgreSQL fork.


### [10 Tips for Building Resilient Payment Systems](https://yomu.fyi/post/10-tips-for-building-resilient-payment-systems.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 28, 2022
- Type: Explainer

The post distills lessons from five years working on Shopify’s payment infrastructure into guidance for building systems that tolerate failure and large-scale traffic. It frames the advice partly around onboarding developers who were new to payments or Shopify’s scale, and notes that load testing for BFCM 2021 began in July with globally distributed load generation. It recommends low timeouts across HTTP clients and data stores, circuit breakers such as Semian, and capacity reasoning with Little’s Law, rate limiting, and load shedding. Monitoring should cover latency, traffic, errors, and saturation, while structured, machine-readable logs make centralized records searchable; load tests use benchmark gateways that mimic production partner latency and capacity. For incidents, Shopify uses defined roles, Slack-based coordination, and retrospectives that turn incorrect assumptions into safeguards and training material.


### [Implementing "seen by" functionality with Postgres](https://yomu.fyi/post/implementing-seen-by-functionality-with-postgres.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Victor
- Published: Jul 18, 2022
- Type: Benchmark

Tracking unique post views within Postgres presents trade-offs between counter accuracy, row bloat, and concurrent write performance. To evaluate options under an architecture constraint preventing external dependencies, a benchmark suite generated synthetic users, skewed post distributions, and replayed view actions. The benchmark tested four implementations: a naive counter column, an hstore key-value approach, an association table, and HyperLogLog (HLL). Benchmark results demonstrated that simple-hstore achieved the lowest average latency among deduplicating approaches at 2.15 milliseconds, closely followed by HLL at 2.16 milliseconds. Despite hstore's raw performance in prototypes, HLL is recommended for production because it circumvents row bloat as view counts grow while avoiding expensive row counts.


### [Typeahead Search at Nextdoor](https://yomu.fyi/post/typeahead-search-at-nextdoor.md)
- Company: [Nextdoor](https://yomu.fyi/company/nextdoor.md)
- Author: Jerry Tian
- Published: Jul 6, 2022
- Type: Problem & solution

Nextdoor built a proximity-based autocomplete service to power typeahead search and mention features across its hyperlocal platform for hundreds of millions of entities, including users and businesses. The system shards geographic data using Uber's open-source H3 geohashing library and stores prefix indexes in memory using Redis sorted sets. By adopting a Command Query Responsibility Segregation architecture, ingestion writes are processed on Redis primary nodes and replicated to read-only search nodes with under 10 milliseconds of replication lag. Dedicated APIs handle indexing, typeahead lookups, and ranking before returning hydrated results. Operating since August 2021, the service processes hundreds of millions of monthly typeahead queries while maintaining a P95 search latency below 30 milliseconds.


### [Graph Networks - 10X investigation with Graph Visualisations](https://yomu.fyi/post/graph-networks-10x-investigation-with-graph-visualisations.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Fujiao Liu
- Published: Jun 30, 2022
- Type: Problem & solution

Fraud detection traditionally required investigators to manually combine large datasets from disparate anti-fraud systems using statistical methods, which proved slow and inefficient. Grab built an interactive Graph Visualisation platform to transform raw records into connected visual maps without requiring manual queries or switching tools. The platform manages over three billion nodes and edges, allowing investigators to selectively expand data points and replay chronological events using temporal filters. Visual relationship mapping helps teams verify account appeals, uncover device-sharing rings, and spot anti-money laundering behavior through transaction density patterns.


### [Navigating Recurring Payments in India: A Backend Perspective](https://yomu.fyi/post/navigating-recurring-payments-in-india-a-backend-perspective.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 29, 2022
- Type: Explainer

India’s October 2021 Reserve Bank of India (RBI) directive required recurring payments to use payer-authorized e-mandates, Additional Factor of Authentication (AFA), and pre-debit notifications at least 24 hours before charges. Shopify’s billing platform therefore had to model e-mandates, support recurring and one-time payments across cards and Unified Payments Interface (UPI), and integrate a local payment provider. The payment service reused its source object for the e-mandate token, recorded the temporary authorization as a charge for auditability, and refunded it asynchronously. To avoid two consecutive card authentication flows, the team used recurring-payment tokenization for both payment paths, accepting an atypical notification experience as a short-term trade-off; UPI required asynchronous webhook-driven handling and a hard mandate limit. After the beta launch, billing-payment success rates in India increased from pre-project levels, and the team said it would soon make UPI generally available to Indian merchants.


### [The Journey to Cloud Development: How Shopify Went All-in on Spin](https://yomu.fyi/post/the-journey-to-cloud-development-how-shopify-went-all-in-on-spin.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 9, 2022
- Type: Problem & solution

Shopify’s development environment, the Ruby-based dev tool and xhyve-backed railgun VM, no longer handled the growing number of repositories, services, integrations, and resource demands around its monoliths. Developers needed more services, faced sluggish code-build-test loops, and encountered costly state changes when tophatting collaborators’ work. The Environments team concluded that development environments needed cloud-like elasticity, then tested an automatically configured local Kubernetes cluster and easy creation of Google Compute Engine VMs; the source also describes Isospin and Spin as the resulting direction. Local Kubernetes did not gain broad organic traction, while the current Isospin implementation is described as the team’s best iteration for developers working on majestic monoliths. The team plans to evolve it into an application-development library built on dev, add dependency inference, and address the tension between persistent environments and Kubernetes pod relocation.


### [To Thread or Not to Thread: An In-Depth Look at Ruby’s Execution Models](https://yomu.fyi/post/to-thread-or-not-to-thread-an-in-depth-look-at-ruby-s-execution-models.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: May 31, 2022
- Type: Explainer

Threaded servers such as Puma and thread-based Sidekiq are popular Ruby deployment choices because threads can increase throughput without a proportional increase in memory. The post compares process and thread models using a memory model that separates static application memory from per-request processing memory, then refines it with Copy on Write (CoW), where forked processes share pages until writes force copies. It explains why RSS can double-count shared pages, recommends PSS and Linux’s /proc/$PID/smaps\_rollup for more meaningful measurements, and gives a production example in which about two-thirds of static memory is shared. The conclusion is metric-dependent: preload applications to improve CoW, prefer fewer larger containers when appropriate, consider Unicorn for reliable request timeouts, and start web workloads at two threads unless substantial I/O wait justifies more; job processors may benefit from higher counts.


### [Automated Experiment Analysis - Making experimental analysis scalable](https://yomu.fyi/post/automated-experiment-analysis-making-experimental-analysis-scalable.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Albert Cheng
- Published: May 30, 2022
- Type: Problem & solution

Manual ad-hoc analysis of online controlled experiments at Grab introduced operational inefficiencies, inconsistent quality control, and scalability barriers across teams. To resolve these issues, Grab extended its GrabX experimentation platform with an Automated Experiment Analysis system that standardises metrics and automates statistical evaluations. The architecture stores experiment configurations and metric definitions from Cosmos DB into Azure Data Lake as bronze datasets, uses Spark on Databricks via Azure Data Factory to process subjects into silver datasets, and applies an internal Python Decision Engine to generate final gold results. These gold datasets are stored in star-schema fact and dimension tables and presented directly in the GrabX interface using embedded Power BI visualisations. The automation eliminates repetitive data pipeline construction for analysts, ensures reproducible findings aligned with initial hypotheses, and accelerates product launch decisions.


### [How Mike Lyndon is using Supabase to accelerate development of AllPullTogether](https://yomu.fyi/post/how-mike-lyndon-is-using-supabase-to-accelerate-development-of-allpull.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Published: May 26, 2022
- Type: Explainer

A collection of case studies outlines how diverse engineering teams adopt Supabase across various infrastructure, authentication, and database workloads. Companies like Good Tape and Shotgun achieved substantial cost reductions between 60% and 83% by migrating to Supabase managed Postgres and authentication services. For security and data access control, Quilia reduced development time by 75% leveraging Supabase Data API alongside Row Level Security features. In machine learning applications, organizations such as Firecrawl, Berri AI, and Quivr utilize Supabase Vector for PostgreSQL vector embeddings rather than standalone vector databases. Additional migrations include Mobbin transferring 200,000 users from Firebase for authentication, while teams like Replenysh implemented one-time password workflows in under 24 hours.


### [Lessons Learned From Running Apache Airflow at Scale](https://yomu.fyi/post/lessons-learned-from-running-apache-airflow-at-scale.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: May 23, 2022
- Type: Explainer

Shopify describes operating Apache Airflow 2.2 on Kubernetes with the Celery executor and MySQL 8 for data extraction, machine-learning training, Iceberg maintenance, and DBT modeling. In its largest environment, Shopify runs over 10,000 DAGs, averages over 400 tasks running at a given moment, and executes over 150,000 runs per day, while facing slow file access, metadata growth, ownership tracking, DAG-author control, traffic bursts, and resource contention. The authors describe replacing per-pod GCSFuse access with an in-cluster NFS volume synchronized to GCS, retaining GCS as the user-facing source of truth; they also use 28-day metadata retention, a YAML manifest, DAG policies, and scheduling controls including pools, priority weights, and Celery queues. These measures are presented as ways to improve file performance, limit database growth, identify owners, enforce constraints, and reduce workload interference, though the retention policy does not support durable history-dependent features such as long-running backfills.


### [How Sempre Health is leveraging the Expert Acceleration Program to accelerate their ML roadmap](https://yomu.fyi/post/how-sempre-health-is-leveraging-the-expert-acceleration-program-to-acc.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: system
- Published: May 19, 2022
- Type: Problem & solution

Sempre Health handles thousands of inbound patient text messages daily to manage medication adherence and prescription discounts. Although an existing rule-based system caught around 80% of inbound texts, the team required statistical machine learning to improve message parsing and reduce manual workload. Collaborating with Hugging Face through the Expert Acceleration Program, the engineers received guidance on labeling strategies, problem formulation, and model selection. They deployed a natural language processing pipeline into production to automatically classify and respond to incoming messages or initiate automated actions like refill requests. Since deployment, the new NLP system automatically handles nearly 20% of incoming messages that previously required operational support tickets.


### [Double Entry Transition Tables: How We Track State Changes At Shopify](https://yomu.fyi/post/double-entry-transition-tables-how-we-track-state-changes-at-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: May 18, 2022
- Type: Explainer

Shopify needed a reliable way to count merchants using Shopify Balance over time, where usage requires both an active Shopify Balance account and an active Shopify account. The data team built accounts\_transition\_facts with double entry transition tables, a format that represents each state change as a -1 row for the previous state and a +1 row for the new state. Individual attribute tables use account\_id, transition\_at, and an index that breaks timestamp ties; the pipeline merges attributes, carries forward missing values, and generates net\_change rows. Because net\_change is additive, queries can sum it while filtering statuses, group results by date, or feed daily aggregates, while additional attributes can be added without rewriting existing SQL or PySpark. The initial build required substantial effort but produced a reporting structure intended to scale with Shopify Balance’s growing complexity.


### [Shopify Invests in Research for Ruby at Scale](https://yomu.fyi/post/shopify-invests-in-research-for-ruby-at-scale.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: May 16, 2022
- Type: Announcement

Shopify says it has invested nearly half a million dollars over the past year in academic research directed toward Ruby and the needs of its community. The work is intended to make programming-language developments account for Ruby’s metaprogramming, loose C extension API, and large-scale workloads with frequent redeployments, rather than assuming long virtual-machine warmup periods. Collaborators include Laurence Tratt, who is investigating hardware meta-tracing and basic-block stitching to generate a just-in-time compiler from the Ruby interpreter; Steve Blackburn, whose group is putting the MMTk garbage-collection toolkit into Ruby; and Stefan Marr, who is studying interpreter speed and startup and warmup time. The program also connects these researchers with Shopify’s Ruby engineers and complements internal work on YJIT and TruffleRuby, with more research and academic papers planned.


### [Maestro: The Orchestration Language Powering Shopify Flow](https://yomu.fyi/post/maestro-the-orchestration-language-powering-shopify-flow.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: May 13, 2022
- Type: Explainer

Shopify developed Maestro, a domain-specific orchestration language, to decouple Shopify Flow’s merchant-facing visual editor from its execution engine, whose requirements differ in declarative usability versus scalable, fault-tolerant execution. The team introduced a horizontally scalable Maestro engine, translated existing Flow workflows into Maestro, migrated them incrementally, and later connected a new visual language directly to Maestro; all workflows had moved to the new editor and engine by early 2022. Maestro coordinates calls to host-language functions rather than implementing service or database operations, while durable checkpoints capture completed work and let the interpreter recover after crashes with at-least-once semantics for primitive calls. During activation, static usage analysis identifies needed returned attributes for possible query optimization, and compilation removes runtime-irrelevant definitions before deployment; checkpoints also support execution monitoring, while planned work includes richer language features, orchestration merging, and workflow preview and testing.


### [Accelerate Large Model Training using PyTorch Fully Sharded Data Parallel](https://yomu.fyi/post/accelerate-large-model-training-using-pytorch-fully-sharded-data-paral.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Sourab Mangrulkar, Sylvain Gugger
- Published: May 2, 2022
- Type: Benchmark

Increasing parameter counts in modern machine learning models make loading and training them prohibitive on standard hardware. While Distributed Data Parallel replicates the entire model across GPUs, PyTorch Fully Sharded Data Parallel shards optimizer states, gradients, and parameters across data-parallel workers. Benchmarks on causal language modeling using GPT-2 Large and XL across Titan RTX GPUs demonstrate that FSDP increases allowable batch sizes and prevents out-of-memory errors encountered in standard DDP. Integrating FSDP through Hugging Face Accelerate enables these capabilities alongside CPU offloading to run models exceeding GPU memory limits. However, Accelerate requires preparing models before creating optimizers when using multiple models and notes that mixed precision support with FSDP currently has transformer compatibility issues.


### [Graph Networks - Striking fraud syndicates in the dark](https://yomu.fyi/post/graph-networks-striking-fraud-syndicates-in-the-dark.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Muqi Li
- Published: Apr 28, 2022
- Type: Problem & solution

Grab's Integrity team faced escalating threats from organized fraud syndicates operating device farms and GPS spoofing tools that mimic normal user behavior. To overcome the limitations of traditional rule-based systems, Grab built a Graph-based Prediction Platform that links entities across shared attributes to expose hidden network structures. The architecture includes a scalable graph database platform holding over one billion nodes, powering real-time graph visualization for analysts alongside a configurable network feature extraction system. In addition, the team deployed Graph Neural Networks to capture structural correlations across transaction networks. These systems collectively help detect coordinated promo abuse, payment fraud, and complex money laundering schemes.


### [Habana Labs and Hugging Face Partner to Accelerate Transformer Model Training](https://yomu.fyi/post/habana-labs-and-hugging-face-partner-to-accelerate-transformer-model-t.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Susan Lansing
- Published: Apr 12, 2022
- Type: Announcement

Training transformer models across computer vision, speech, and natural language processing tasks at scale often demands heavy compute resources, incurring significant time and financial expense. To address these bottlenecks, Habana Labs and Hugging Face partnered to integrate the SynapseAI software suite into the Hugging Face Optimum open-source library. This integration allows machine learning practitioners to accelerate transformer training workflows on Habana Gaudi processors using minimal code adjustments. Habana Gaudi hardware, featured in Amazon EC2 DL1 instances and Supermicro X12 servers, incorporates ten 100 Gigabit Ethernet ports per processor to scale from single units to thousands of chips. The combined hardware and software architecture supports TensorFlow and PyTorch while delivering price and performance metrics up to 40% lower than comparable training alternatives.


### [The Magic of Merlin: Shopify's New Machine Learning Platform](https://yomu.fyi/post/the-magic-of-merlin-shopify-s-new-machine-learning-platform.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Apr 6, 2022
- Type: Explainer

Shopify’s machine learning platform team redesigned its infrastructure to accommodate varied use cases, dependencies, data types, integrations, and production requirements. Merlin uses an open-source stack in which on-demand Merlin Workspaces run as short-lived Ray clusters on Kubernetes, while projects isolate code and dependencies in Docker-based environments. Its first iteration focuses on training and batch inference, with users moving from local or Jupyter-based prototypes to automated production workflows through the Merlin API, Airflow, or Oozie. Shopify validated the platform by migrating its product categorization workflows, using Ray Train for distributed TensorFlow training and Ray ActorPool for batch inference; planned extensions include online inference, model lifecycle management, and machine learning monitoring.


### [Supabase Realtime, with Multiplayer Features](https://yomu.fyi/post/supabase-realtime-with-multiplayer-features.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Wen Bo Xie
- Published: Apr 1, 2022
- Type: Announcement

Supabase announced the third major release of Supabase Realtime, evolving the system from a single-tenant PostgreSQL change listener into a multi-tenant, distributed real-time platform. Built using Elixir and the Phoenix framework across a distributed node cluster, the new architecture decouples the database listener into an extension model and opens generic WebSocket functionality. New multiplayer capabilities include Presence, which uses Conflict-free Replicated Data Types (CRDTs) to track user connection states, and Broadcast, which distributes ephemeral messages across topics without writing to disk. The modular extension design allows listening to multiple PostgreSQL instances, external systems like MySQL, and financial or blockchain feeds, as demonstrated on a globally distributed 20-node cluster.


[Newer posts](https://yomu.fyi/topic/scalability/page/14.md) · [Older posts](https://yomu.fyi/topic/scalability/page/16.md)