Loading…
Topic
372 posts about Scalability. Every summary links to the original.
See the idea
Consolidating agent memory, retrieval, and governance onto MongoDB Atlas removes sync and vendor layers, but ties scaling and control to one platform. This trade-off view shows the payoff, the cost, and the condition that decides it.
Autoscaling trades responsiveness, stability, and cost: this explainer shows how signal choice, scaling direction, and capacity limits shape the right control loop for a workload.
Apache Pinot is a distributed real-time OLAP datastore for low-latency analytics. This architecture view shows how streams, processing, partitioning, indexed storage, and replica-aware serving fit together.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Search at Shopify is presented as a case for range across data science and engineering, rather than narrow specialization in either discipline. The article describes a team that treats work as “search” instead of separating data and engineering responsibilities: engineers learn experimentation and model evaluation, while data scientists build performant, testable, maintainable production systems. It frames search as a sequence of trade-offs among relevance, correctness, scalability, performance, stability, and maintainability, arguing that people who understand only half the constraints can produce siloed handoffs, deployment delays, or suboptimal systems. Shopify’s stated model is a single collaborative team with shared planning and execution, intended to reduce territorial boundaries and support smart, fast, scalable search. The article extends this principle to personalized and conversational products, concluding that future work will require practitioners to move between data and engineering.
Shopify’s 2021 BFCM live map had to scale from more than 1.7 million merchants while adding product trends and unique shoppers, after the prior Cricket-and-Redis backend showed volume and delivery weaknesses. The redesign placed Apache Flink on the critical path to filter irrelevant checkout events, reducing Cricket’s workload to about one percent of event volume, while cross-region sharding and active-active deployment supported availability. Flink computed new metrics, including 500 product categories ranked by sales-volume change every five minutes, while Cricket retained existing metric computation and deduplication. In load tests, the pipeline handled 1 million checkout events per second, and its jobs ran at 100 percent uptime during BFCM without backpressure or manual intervention. Redis still caused high CPU and memory pressure, producing brief arc-visual errors for a small sample of users, so Shopify plans a simpler Flink-centered design.
In September 2021, Shopify completed its cutover from Percona Server 5.7.21 to 5.7.32 across a MySQL fleet spanning hundreds of horizontally sharded databases and thousands of virtual machines. The upgrade targeted replication improvements from transaction writesets, along with security and stability fixes, while avoiding the larger risks of a major-version change. The Database Platform team extended Mason into a declarative topology system and used Chef roles to provision upgraded VMs, restore backups, run mysql_upgrade when required, configure replication, and populate replacement topologies. Extensive rollback testing exposed a critical downgrade bug; after patching 5.7.21, Shopify proceeded in staggered batches beginning August 25 and completed the final shards on September 7, while documenting a repeatable process intended to make future upgrades faster and safer.
Continuous ride-hailing assignment relies on solving the minimum weight bipartite matching problem between passengers and driver-partners. While traditional implementations assume a precalculated cost matrix, computing shortest-path travel times across large road networks dominates total execution time. Researchers introduced an Incremental Kuhn-Munkres algorithm that leverages the spatial locality of optimal matches to compute edge costs on demand. The approach integrates priority queues and lower-bounding techniques with refinement rules to avoid evaluating distant pairs while guaranteeing the same optimal assignment. Evaluated on Singapore road network data and real Grab production workloads, the incremental techniques reduced exact cost calculations and decreased assignment running times by over an order of magnitude.
Supabase announced a $30 million Series A funding round led by Coatue, bringing its total funding to $36 million. The company aims to provide an open source alternative to Firebase while addressing the scalability limitations typical of abstracted database platforms. Rather than building a compatibility layer, Supabase provisions a full PostgreSQL database for every project alongside integrated authentication, storage, RESTful APIs, and realtime data streaming. Platform features rely directly on core PostgreSQL primitives, including Row Level Security policies, database triggers, and logical replication streams. Funding will support long-term development of a cloud-native PostgreSQL platform offering database branching, distributed read-write nodes, scalable storage, and ephemeral compute that scales down to zero.
The 2021 AI4TSP Competition addressed the Travelling Salesperson Problem when travel times are sampled from probability distributions and locations carry rewards and time windows. With n locations, the number of possible tours is n!, making exhaustive enumeration impractical, while the competition also required decisions under uncertain travel outcomes. One track required a fixed tour optimized for expected performance across scenarios; the other allowed an adaptive policy to select the next location as time elapsed, with results evaluated across many instances and realizations. The Shopify-led team used dialectic search for the fixed-tour track and deep reinforcement learning for the on-the-fly track, and won first prize in both.
Shopify’s MySQL shards can become unevenly utilized as merchants grow after being assigned to arbitrary shards, increasing over-utilization risk while leaving other capacity underused. The post describes a strategy for selecting shop placements using historical utilization and traffic data, simulations, and potentially other metrics such as shop size, GMV, and move time. It then explains online shop migration with Ghostferry, which batch-copies records and tails MySQL’s binary log before a brief cutover updates the routing table and directs new writes to the target pod. Verification checks data completeness, integrity, and stale traffic before old data is pruned; the migration system’s central algorithm is also represented in a TLA+ formal specification. The stated outcome is a more balanced infrastructure that can isolate high-traffic merchants across pods with virtually zero consumer-facing downtime.
Shop’s home feed, the app’s most used feature, aggregated orders from millions of Shopify and non-Shopify merchants plus tracking data from dozens of carriers, accounted for 30% of database load and affected application performance. Before each database update, the custom write-through cache marks a user’s cache invalid, applies the write, then refreshes and revalidates it. To handle concurrent updates, it uses a separate pending-writes key and a short-expiry key to distinguish active writes from interrupted processes, while Active Record Concerns preserve the existing ORM interface. After a staged validation rollout, the cache reduced database load by 15%, overall app latency by about 20%, and overall GraphQL latency by 20%, with a double-digit decrease in CPU usage.
Shopify revisited product categorization as its platform expanded to millions of merchants selling billions of products across diverse industries. The model uses product titles, descriptions, types, vendors, collections, tags, and images, converting them into embeddings with Multilingual BERT and MobileNet-V2 before hierarchical classification across the seven-level Google Product Taxonomy. Each taxonomy level is treated as a separate multi-class task, with parent predictions fed into child predictions and weighted losses combined during training; confidence thresholds filter predictions at each level. The upgraded system increased leaf precision by 8% while almost doubling coverage, although thresholds require balancing accuracy, coverage, merchant experience, and performance in sensitive categories, and future work includes addressing data imbalance and adding merchant-level features.
Supabase introduced a dedicated Reports section and updated project home pages to deliver observability across hosted Postgres projects and complementary services. Because developers can run unconstrained database queries and custom workloads, monitoring both project-level service usage and underlying infrastructure metrics is essential. The new dashboard displays one month of historical data, including CPU and memory usage alongside request volumes for Database, Auth, and Storage APIs. To collect metrics across hundreds of thousands of targets without high latency, Supabase implemented a federated monitoring architecture running regional Prometheus instances that scrape local exporters and aggregate subsets into a central VictoriaMetrics instance. All project databases now also launch with the pg_stat_statements extension enabled for future query statistics.
Mobbin originally launched its mobile design search platform using Firebase, quickly growing to hundreds of thousands of registered users. As the platform scaled, the team faced user friction because Firebase Authentication did not merge accounts sharing the same email address across different providers. Furthermore, mutative Firebase Function failures threatened document data integrity, and multi-round-trip document queries drove high monthly costs across five million API requests. To resolve these challenges, Mobbin migrated its backend to Supabase, adopting its GoTrue authentication service and underlying Postgres relational database. This migration resolved authentication conflicts, safeguarded data consistency, and reduced monthly infrastructure expenses since Supabase does not charge on a per-API-request basis.
Shopify describes a playbook for running large engineering programs as platform complexity grows across products, tooling, and architecture. The framework begins with a Program Plan covering the problem statement, objectives, guiding principles, definition of done, risks, mitigation, staffing, scope, and a path to completion. Execution uses six-week cycles that revisit unfinished goals, regressions, emerging risks, and the planned route to completion; the described program spans 200 people, nine sub-organizations, and 92 projects. Supporting rituals include weekly updates, status checks, risk and escalation triage, cycle reviews, retrospectives, RFCs, and performance testing with Lua scripts orchestrated by Genghis. The post concludes that engineering program management must adapt to each organization and that a combination of scheduled and ad hoc rituals helped Shopify pursue its goals.
Pretraining state-of-the-art Transformer language models typically requires vast computing resources and expensive hardware clusters that are inaccessible to many researchers. Standard distributed deep learning struggles over public Internet infrastructure because consumer connections suffer from limited bandwidth, dynamic peer disconnections, and restrictive firewalls. To resolve these networking bottlenecks, Distributed Deep Learning in Open Collaborations (DeDLOC) introduces an adaptive training framework that accumulates massive batch sizes across heterogeneous volunteer devices before applying optimizer updates. The framework splits gradient vectors dynamically based on participant connection speeds and allows firewalled nodes to contribute data without performing aggregation. In a practical deployment with 40 volunteers, the 18-million-parameter sahajBERT model achieved Bengali downstream benchmark performance comparable to much larger multilingual models trained on dedicated supercomputers.
Shopify’s Trino-based interactive query infrastructure faced inconsistent latency and operational failures as workloads grew to 15 Gbps and more than 300 million rows per second. The team analyzed query volumes, resource-heavy datasets, CPU wall time, and failure scenarios, then used a Trino Query Replicator to reproduce historical degradation and isolate issues involving compressed Kafka JSON, shared cluster classes, resource allocation, JVM settings, and dataset statistics. They added workload-specific and ephemeral clusters, reduced concurrent queries, applied recommended JVM recompilation settings, and limited task drivers to 16; they also scaled worker pods to 61 cores and 220 GB memory, although scaling alone was unstable. The infrastructure changes produced a stable environment and reduced P95 execution time to below 30 seconds, moving more queries into the zero-to-five-second range, while the five-second target remained a future goal.
Unpredictable traffic spikes, slow third-party payment requests, and resource-intensive image processing can overwhelm an application, making it slow or unavailable; the text presents background jobs as a way to preserve meaningful user interactions. High availability is defined not merely as server uptime, but as users being able to interact with the application when needed. The application server places a task message in a queue and continues serving requests, while workers process jobs asynchronously, with support for retries, parallelization, prioritization, and event- or time-based scheduling. Offloading improves response times and buffers spikes, but changes immediate guarantees: an upload may be accepted before processing succeeds, and errors may be handled later. For distributed systems, data-based messaging such as RabbitMQ can connect different services, while Kafka's append-only, replayable event log supports shared consumption and event sourcing.
Shopify describes how its SSL team moved merchants’ stores to HTTPS and ultimately secured more than 4.5 million domains with Let’s Encrypt. The effort began against certificate providers whose APIs were not designed for fully automated provisioning: errors often arrived as human-readable messages, and imposed throttling could have stretched initial issuance to about 100 days. Shopify tested the ACME protocol, valued its standardized specification and open-source implementation, and first added Let’s Encrypt as a backup certificate authority to improve emergency key or certificate-chain rotation. The team rolled out that integration in a few hours, compared with months for earlier providers, then made Let’s Encrypt its primary authority after finding it reliable. The account presents this change as evidence that ACME can make certificate automation more responsive and allow certificate authorities to be added without redesigning Shopify’s infrastructure.
Shopify’s GraphQL Admin API uses calculated query costs to address two limitations of request-based rate limiting: identical credits for responses of different sizes and equal treatment of reads and side-effecting writes. The server statically analyzes each query, assigning one point to objects and interfaces or unions, zero to scalars and enums, two plus the requested object count to connections, and 10 to mutations. Clients receive 50 points per second up to a 1,000-point limit, while responses expose requested and actual costs; when fewer connection records are returned, the difference is refunded. The post reports a linear correlation between calculated query complexity and execution time, giving Shopify more predictable load for infrastructure scaling and a way to identify performance outliers.
GraphQL clients can face slower, larger queries and difficult roll-outs as product screens gain features and data. The Orders & Fulfillments teams examined these issues while targeting a sub-one-second mobile page load on a reliable network. Starting with a product-list query that loads 100 products, the post recommends pagination with page-size/index controls and a hasNextPage field, plus performance tripwires to track loading time. It describes @include and @skip directives, runtime query construction, and chained queries for feature flags or parameters, while warning that chained requests add latency. For growing screens, it separates unrelated filters, permissions, and banners into parallel queries, enabling independent scaling, partial rendering, and less redundant pagination work, while cautioning that server load and partial-data state require monitoring and possible refactoring.
Shopify describes how its Privacy and Data Science & Engineering teams redesigned analytical event handling to make personal-information deletion workable at Shopify’s scale. The effort addressed a lack of guaranteed event schemas by introducing a human-readable JSON schematization system covering field types, version evolution, ownership, data controllers and subjects, and privacy instructions; more than 4,500 active schemas now govern events. Before warehouse storage, identifying data is either obfuscated into generalized values, such as masked IP addresses with city or country information, or tokenized into consistent random values whose mappings reside in a secured vault and vary by controller and subject. Removing a vault mapping makes warehouse copies of the token non-detokenizable, allowing deletion without warehouse operations; the production platform processes about 20 billion events daily, while tens to hundreds of millions of mappings are deleted daily.
Scaling BERT-like model inference on CPUs presents practical deployment challenges as transformer architectures proliferate in production environments. Using a reproducible Hydra-based benchmarking framework, inference workloads were evaluated on an Amazon Web Services c5.metal instance featuring an Intel Xeon Platinum 8275 processor with AVX512 and VNNI support. Out-of-the-box evaluations showed PyTorch 1.8.1 delivering lower latency than Google TensorFlow 2.4.0, likely due to PyTorch relying on OpenMP and Intel MKL/oneDNN rather than Eigen. Implementing multiple inference streams by pinning parallel model instances to dedicated CPU cores yielded near-linear throughput scaling. Matching instance core allocations to specific problem sizes also demonstrated that smaller workloads can achieve optimal latency on fewer cores, enabling up to a sixfold reduction in cloud compute costs.
The best engineering reads, once a week.
