---
title: "Scalability"
description: "379 posts about Scalability, summarised, each linking to the original."
---

# Scalability
> 379 posts about Scalability, summarised, each linking to the original.

## Articles

### [Shopify's Path to a Faster Trino Query Execution: Infrastructure](https://yomu.fyi/post/shopify-s-path-to-a-faster-trino-query-execution-infrastructure.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 9, 2021
- Type: Problem & solution

Shopify’s Trino-based interactive query infrastructure faced inconsistent latency and operational failures as workloads grew to 15 Gbps and more than 300 million rows per second. The team analyzed query volumes, resource-heavy datasets, CPU wall time, and failure scenarios, then used a Trino Query Replicator to reproduce historical degradation and isolate issues involving compressed Kafka JSON, shared cluster classes, resource allocation, JVM settings, and dataset statistics. They added workload-specific and ephemeral clusters, reduced concurrent queries, applied recommended JVM recompilation settings, and limited task drivers to 16; they also scaled worker pods to 61 cores and 220 GB memory, although scaling alone was unstable. The infrastructure changes produced a stable environment and reduced P95 execution time to below 30 seconds, moving more queries into the zero-to-five-second range, while the five-second target remained a future goal.


### [High Availability by Offloading Work Into the Background](https://yomu.fyi/post/high-availability-by-offloading-work-into-the-background.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 8, 2021
- Type: Explainer

Unpredictable traffic spikes, slow third-party payment requests, and resource-intensive image processing can overwhelm an application, making it slow or unavailable; the text presents background jobs as a way to preserve meaningful user interactions. High availability is defined not merely as server uptime, but as users being able to interact with the application when needed. The application server places a task message in a queue and continues serving requests, while workers process jobs asynchronously, with support for retries, parallelization, prioritization, and event- or time-based scheduling. Offloading improves response times and buffers spikes, but changes immediate guarantees: an upload may be accepted before processing succeeds, and errors may be handled later. For distributed systems, data-based messaging such as RabbitMQ can connect different services, while Kafka's append-only, replayable event log supports shared consumption and event sourcing.


### [Let’s Encrypt x Shopify: Securing the Web 4.5 Million Domains at a Time](https://yomu.fyi/post/let-s-encrypt-x-shopify-securing-the-web-4-5-million-domains-at-a-time.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 24, 2021
- Type: Explainer

Shopify describes how its SSL team moved merchants’ stores to HTTPS and ultimately secured more than 4.5 million domains with Let’s Encrypt. The effort began against certificate providers whose APIs were not designed for fully automated provisioning: errors often arrived as human-readable messages, and imposed throttling could have stretched initial issuance to about 100 days. Shopify tested the ACME protocol, valued its standardized specification and open-source implementation, and first added Let’s Encrypt as a backup certificate authority to improve emergency key or certificate-chain rotation. The team rolled out that integration in a few hours, compared with months for earlier providers, then made Let’s Encrypt its primary authority after finding it reliable. The account presents this change as evidence that ACME can make certificate automation more responsive and allow certificate authorities to be added without redesigning Shopify’s infrastructure.


### [Rate Limiting GraphQL APIs by Calculating Query Complexity](https://yomu.fyi/post/rate-limiting-graphql-apis-by-calculating-query-complexity.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 22, 2021
- Type: Explainer

Shopify’s GraphQL Admin API uses calculated query costs to address two limitations of request-based rate limiting: identical credits for responses of different sizes and equal treatment of reads and side-effecting writes. The server statically analyzes each query, assigning one point to objects and interfaces or unions, zero to scalars and enums, two plus the requested object count to connections, and 10 to mutations. Clients receive 50 points per second up to a 1,000-point limit, while responses expose requested and actual costs; when fewer connection records are returned, the difference is refunded. The post reports a linear correlation between calculated query complexity and execution time, giving Shopify more predictable load for infrastructure scaling and a way to identify performance outliers.


### [Querying Strategies for GraphQL Clients](https://yomu.fyi/post/querying-strategies-for-graphql-clients.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 15, 2021
- Type: Tutorial

GraphQL clients can face slower, larger queries and difficult roll-outs as product screens gain features and data. The Orders & Fulfillments teams examined these issues while targeting a sub-one-second mobile page load on a reliable network. Starting with a product-list query that loads 100 products, the post recommends pagination with page-size/index controls and a hasNextPage field, plus performance tripwires to track loading time. It describes @include and @skip directives, runtime query construction, and chained queries for feature flags or parameters, while warning that chained requests add latency. For growing screens, it separates unrelated filters, permissions, and banners into parallel queries, enabling independent scaling, partial rendering, and less redundant pagination work, while cautioning that server load and partial-data state require monitoring and possible refactoring.


### [Deleting the Undeletable: Managing PII at Shopify - Shopify](https://yomu.fyi/post/deleting-the-undeletable-managing-pii-at-shopify-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 10, 2021
- Type: Problem & solution

Shopify describes how its Privacy and Data Science & Engineering teams redesigned analytical event handling to make personal-information deletion workable at Shopify’s scale. The effort addressed a lack of guaranteed event schemas by introducing a human-readable JSON schematization system covering field types, version evolution, ownership, data controllers and subjects, and privacy instructions; more than 4,500 active schemas now govern events. Before warehouse storage, identifying data is either obfuscated into generalized values, such as masked IP addresses with city or country information, or tokenized into consistent random values whose mappings reside in a secured vault and vary by controller and subject. Removing a vault mapping makes warehouse copies of the token non-detokenizable, allowing deletion without warehouse operations; the production platform processes about 20 billion events daily, while tens to hundreds of millions of mappings are deleted daily.


### [Scaling-up BERT Inference on CPU (Part 1)](https://yomu.fyi/post/scaling-up-bert-inference-on-cpu-part-1.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Morgan Funtowicz
- Published: Apr 20, 2021
- Type: Benchmark

Scaling BERT-like model inference on CPUs presents practical deployment challenges as transformer architectures proliferate in production environments. Using a reproducible Hydra-based benchmarking framework, inference workloads were evaluated on an Amazon Web Services c5.metal instance featuring an Intel Xeon Platinum 8275 processor with AVX512 and VNNI support. Out-of-the-box evaluations showed PyTorch 1.8.1 delivering lower latency than Google TensorFlow 2.4.0, likely due to PyTorch relying on OpenMP and Intel MKL/oneDNN rather than Eigen. Implementing multiple inference streams by pinning parallel model instances to dedicated CPU cores yielded near-linear throughput scaling. Matching instance core allocations to specific problem sizes also demonstrated that smaller workloads can achieve optimal latency on fewer cores, enabling up to a sixfold reduction in cloud compute costs.


### [PgBouncer is now available in Supabase](https://yomu.fyi/post/pgbouncer-is-now-available-in-supabase.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Angelico de los Reyes
- Published: Apr 2, 2021
- Type: Announcement

Serverless JavaScript frameworks and developer tools frequently establish separate database connections during bursty traffic, rapidly exhausting PostgreSQL connection limits in the absence of traditional middleware. To resolve connection surges, Supabase integrated the open-source connection pooler PgBouncer directly onto the PostgreSQL server across all newly created projects. Rather than increasing the total number of connections PostgreSQL can open, PgBouncer recycles open connections and queues excess requests until active connections become available. Developers can manage pooling through the dashboard across Session, Transaction, and Statement modes, with Transaction mode recommended for serverless functions despite disabling session features like prepared statements. The default pool size is initially configured to 15 connections.


### [How Grab Leveraged Performance Marketing Automation to Improve Conversion Rates by 30%](https://yomu.fyi/post/how-grab-leveraged-performance-marketing-automation-to-improve-convers.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Sc Ng
- Published: Mar 22, 2021
- Type: Problem & solution

Grab faced operational bottlenecks managing direct-response Google Ads campaigns across thousands of ad groups due to its hyperlocal marketing across Southeast Asian markets. To eliminate the manual burden of tracking and updating ad creatives, the team built CARA, an in-house automation tool deployed on AWS serverless compute. CARA utilizes standardized file naming conventions to map assets to specific campaigns and connects with Google Ads and YouTube APIs to detect and replace low-performing assets. During an experimental rollout across more than 8,000 active ad groups, CARA replaced nearly 2,000 underperforming creatives. The automated asset replacement workflow produced an 18% to 30% increase in clickthrough and conversion rates.


### [Hugging Face Reads, Feb. 2021 - Long-range Transformers](https://yomu.fyi/post/hugging-face-reads-feb-2021-long-range-transformers.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Victor Sanh
- Published: Mar 9, 2021
- Type: Explainer

Standard transformer models face severe memory and compute bottlenecks due to the quadratic scaling of self-attention with sequence length. To scale beyond short sequence limits, efficient architectures introduce varied mechanisms such as custom sparse attention patterns, compressed recurrence, low-rank projections, and kernel approximations. Longformer integrates local dilated windowed attention with task-specific global tokens, functioning as a drop-in replacement that enables standard pre-trained models to process long inputs without full retraining. Concurrently, Compressive Transformers extend temporal memory by compressing discarded activations, while models like Linformer and Performer reduce computational complexity through projection and kernel estimation. Progress in long-range modeling also reveals that incrementally training models from short to longer sequences accelerates convergence and improves downstream performance.


### [Toad, a link shortener with simple APIs for low-coders](https://yomu.fyi/post/toad-a-link-shortener-with-simple-apis-for-low-coders.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Published: Mar 8, 2021
- Type: Explainer

Supabase compiles an overview of customer case studies documenting how organizations deploy its platform across production environments. Multiple companies report infrastructure migrations, including Firecrawl switching from Pinecone to Supabase Vector for PostgreSQL vector embeddings and Mobbin moving 200,000 users from Firebase. Engineering teams document operational gains and cost efficiencies, such as Shotgun achieving an 83% reduction in data infrastructure expenses and Quilia saving 75% in development time using Data API and RLS features. Other organizations highlight infrastructure performance improvements, including Good Tape recording a 60% cost reduction alongside database efficiency gains after migrating to Supabase managed Postgres and Authentication. Teams also report accelerated implementation speeds, with Replenysh delivering OTP functionality in less than 24 hours and Xendit releasing a full solution to production within one week.


### [Retrieval Augmented Generation with Huggingface Transformers and Ray](https://yomu.fyi/post/retrieval-augmented-generation-with-huggingface-transformers-and-ray.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: system
- Published: Feb 10, 2021
- Type: Problem & solution

Huggingface Transformers added the Retrieval Augmented Generation architecture to combine standard sequence-to-sequence models with external document retrieval for knowledge-intensive tasks. Scaling RAG fine-tuning across multiple GPUs previously created bottlenecks because the torch.distributed implementation forced the single rank 0 worker to handle all retrieval index lookups. To resolve this synchronization bottleneck, Ray was integrated into the contextual document retrieval mechanism using stateful actor abstractions. Dedicated Ray processes separate from the training workers now load the large index and process retrieval requests concurrently. This architectural separation delivers up to a 2x speedup per retrieval call and removes the strict dependency on PyTorch for training.


### [Roboflow.com choose Supabase to power Paint.wtf leaderboard](https://yomu.fyi/post/roboflow-com-choose-supabase-to-power-paint-wtf-leaderboard.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Rory Wilding
- Published: Feb 9, 2021
- Type: Problem & solution

Brad Dwyer and Erik Dunteman created Paint.wtf to experiment with OpenAI's CLIP model, allowing users to draw images evaluated against text prompts and scored on a leaderboard. While the team initially considered Firebase, the platform lacked built-in document counting functionality essential for their leaderboard architecture. To avoid implementing custom workarounds or exporting data to external systems, they selected Supabase to leverage PostgreSQL's native counting capabilities. After launching on Hacker News, Reddit, and Product Hunt, the platform experienced viral traffic that peaked at over two drawing submissions per second and surpassed one hundred thousand participating users within twenty-four hours. The PostgreSQL-backed infrastructure maintained reliable performance throughout the surge, delivering accurate and up-to-date score rankings under heavy load.


### [Customer Support Workforce Routing](https://yomu.fyi/post/customer-support-workforce-routing.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Suman Anand
- Published: Feb 5, 2021
- Type: Problem & solution

Grab replaced its third-party customer support routing software with an in-house workforce routing system for Livechat to gain better priority controls, bespoke configurations, and deeper analytics. The platform separates requests into distinct priority and business queues, using parallel workers that spend varied time slices dequeuing higher-priority issues like safety concerns. To prevent request starvation, workers operate out of sync across queue priority levels while dynamic queue limits cap incoming volume based on agent availability and performance. The system routes requests through an intermediate Agent Group layer, calculating eligibility scores from proficiency and concurrency metrics while managing per-agent locks to prevent over-allocation.


### [Serving Driver-partners Data at Scale Using Mirror Cache](https://yomu.fyi/post/serving-driver-partners-data-at-scale-using-mirror-cache.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Indrajit Sarkar
- Published: Jan 26, 2021
- Type: Problem & solution

Grab's Drivers Data service handles up to 10,000 requests per second during peak hours to supply driver information across backend microservices. The original setup used MySQL with Redis and standalone in-memory local caches, but yielded a low 25% local cache hit rate due to traffic patterns characterized by high burst frequency for individual drivers alongside redundant database calls across nodes. To solve this, the team developed Mirror Cache, an in-memory caching system that pairs Dgraph's Ristretto library with an asynchronous gRPC replication layer to mirror updates across cluster nodes. The replicator batches updates within the same AWS availability zone and forwards data to single nodes across zones to minimize transfer overhead. Production deployment increased the in-memory cache hit rate to approximately 75% and reduced direct MySQL queries by 5%.


### [The GrabMart Journey](https://yomu.fyi/post/the-grabmart-journey.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Clarisse Peralta
- Published: Jan 18, 2021
- Type: Problem & solution

During the COVID-19 pandemic, Grab accelerated the rollout of its GrabMart on-demand grocery delivery service to meet surging regional demand. Early pilot testing within the existing GrabFood interface revealed that grocery shopping fundamentally diverges from meal ordering, as users search at an item level rather than browsing merchants. To support grocery workflows, the team restructured the user experience with item and store categories, one-click add-to-cart functionality, and scheduled delivery options. Engineering teams simultaneously updated core infrastructure across order management, driver allocation systems, and merchant Partner APIs to handle specialized mart bookings. Rollouts were validated across two main phases using the internal Grab Early Access program to catch issues before regional release.


### [Trident - Real-time Event Processing at Scale](https://yomu.fyi/post/trident-real-time-event-processing-at-scale.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Jie Zhang
- Published: Jan 13, 2021
- Type: Problem & solution

Trident serves as Grab's internal real-time event-processing and workflow automation engine, driving user campaigns, rewards, and notifications across multiple business lines. To handle peak loads exceeding 2,000 events per second without duplicate execution, the system consumes decoupled Kafka streams and enforces exactly-once semantics using Redis and MySQL deduplication checks. Processing efficiency relies on server autoscaling aligned with Kafka partition counts, combined with dynamic goroutine allocation per consumer. To minimize rule evaluation overhead, Trident indexes active campaigns into an in-memory hash map by event type, cutting processing time by at least 90%. Furthermore, condition evaluation is optimized through lazy loading and a weighted sorting algorithm that checks low-cost in-memory data prior to executing expensive database queries or external service calls.


### [Pharos - Searching Nearby Drivers on Road Network at Scale](https://yomu.fyi/post/pharos-searching-nearby-drivers-on-road-network-at-scale.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Hao Wu
- Published: Dec 22, 2020
- Type: Problem & solution

Ride-hailing allocation requires identifying nearby drivers based on actual routing distance and estimated time of arrival rather than straight-line haversine distance. Grab developed Pharos, a distributed in-memory spatial microservice that evaluates road networks to match fast-moving drivers with pick-up requests. The system partitions OpenStreetMap road graphs by city and vehicle vertical while indexing driver positions and edge-based nodes using Adaptive Radix Trees. To locate candidates, Pharos projects driver coordinates onto road segments as phantom nodes and runs Incremental Network Expansion to compute isochrone reaches. Read-write concurrency is managed by switching tree root snapshots, allowing concurrent reads while updates produce new roots for subsequent lookups.


### [Resiliency Planning for High-Traffic Events](https://yomu.fyi/post/resiliency-planning-for-high-traffic-events.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 10, 2020
- Type: Explainer

Shopify’s preparation for Black Friday Cyber Monday (BFCM) centers on building and maintaining resiliency practices for a high-traffic platform and large development teams. Regular load and stress tests verify required capacity, probe limits, detect regressions, and compare behavior across differently configured shops. Teams document expected user experiences and dependency failure modes in a resiliency matrix, then use game days to test that model against reality, including alerts, paging, operator readiness, and response procedures. Incident analysis treats disruptions as learning opportunities rather than stopping at a single “root cause”; nearer to BFCM, Shopify slows feature and service changes while increasing work on performance, reliability, scale, runbooks, tools, and bottlenecks. The source says these practices support a platform that powered over $5.1 billion in sales.


### [How to Reliably Scale Your Data Platform for High Volumes](https://yomu.fyi/post/how-to-reliably-scale-your-data-platform-for-high-volumes.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 8, 2020
- Type: Problem & solution

Shopify’s data platform prepared for BFCM 2020, when merchant sales exceeded $5.1 billion and platform throughput rose by an average of 150 percent. The Data Platform Engineering team focused on throughput risk, which grows with volume and is greatest near ingestion and processing, while prioritizing work through four service tiers. For Tier 1 and Tier 2 systems, engineers defined each service’s primary objective, identified adjustable controls such as job frequency, overlap intervals, CPU and memory requests, BigTable node pools, and ProxySQL connection queues, then used controlled load tests to expose capacity and recovery risks. The resulting work included cluster scaling, load-balancing adjustments, query and job tuning, and improved indices, schemas, dashboards, playbooks, SLOs, incident handling, and alerts. The post reports that BFCM 2020 completed without merchant-impacting incidents; only a handful of minor, self-healing incidents occurred, while monitoring and occasional tuning preserved spare capacity.


[Newer posts](https://yomu.fyi/topic/scalability/page/16.md) · [Older posts](https://yomu.fyi/topic/scalability/page/18.md)