---
title: "Scalability"
description: "379 posts about Scalability, summarised, each linking to the original."
---

# Scalability
> 379 posts about Scalability, summarised, each linking to the original.

## Articles

### [Modernizing Core Insurance Systems: Breaking the Batch Bottleneck](https://yomu.fyi/post/modernizing-core-insurance-systems-breaking-the-batch-bottleneck.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Vinod Bagal, Jagpreet Singh
- Published: Sep 18, 2025
- Type: Problem & solution

Migrating core insurance platforms from legacy relational databases with PL/SQL to Java and MongoDB Atlas frequently degrades batch processing performance. Like-for-like migrations can cause batch jobs to run 25 to 30 times slower due to high network round-trips, inefficient per-record operations, and under-utilized bulk capabilities. To resolve these bottlenecks, an optimization framework was developed that leverages MongoDB native bulkWrite operations, intelligent reference data prefetching, and parallel execution. An architecture combining a Spring Boot controller and a dedicated executor framework partitions workloads across threads or event processors. This framework recovered lost performance, completing previously failing batch jobs within service-level agreements and achieving 10 to 15 times better execution speeds than legacy systems in certain workloads.


### [MongoDB.local NYC 2025: Defining the Ideal Database for the AI Era](https://yomu.fyi/post/mongodb-local-nyc-2025-defining-the-ideal-database-for-the-ai-era.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Dev Ittycheria, President and CEO, MongoDB
- Published: Sep 18, 2025
- Type: Announcement

At MongoDB.local NYC, MongoDB introduced new database capabilities, AI integrations, and modernization tools designed for agentic and generative AI workloads. The release of MongoDB 8.2 arrives alongside public previews of Search and Vector Search for both MongoDB Community Edition and Enterprise Server. MongoDB also highlighted Voyage AI embedding models and rerankers to improve retrieval accuracy across raw data, metadata, and embeddings. To address the costs and constraints of legacy infrastructure, the new MongoDB Application Modernization Platform combines AI-assisted tooling and specialized expertise to migrate legacy systems. According to MongoDB, early migrations using this platform run two to three times faster, while code rewriting tasks accelerate by an order of magnitude.


### [MongoDB.local NYC 2025 : définir la base de données idéale à l'ère de l'IA](https://yomu.fyi/post/mongodb-local-nyc-2025-de-finir-la-base-de-donne-es-ide-ale-a-l-e-re-d.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Dev Ittycheria, President and CEO, MongoDB
- Published: Sep 18, 2025
- Type: Announcement

At the MongoDB.local NYC 2025 event, MongoDB unveiled several platform updates aimed at modernizing enterprise database architectures and supporting emerging agentic artificial intelligence workloads. The newly released MongoDB 8.2 version provides enhanced performance alongside expanded developer features across the broader document database ecosystem. The company also integrated Voyage AI embedding and reranking models to improve context retrieval precision and efficiency when building production-grade AI systems. Furthermore, dedicated Search and Vector Search capabilities have reached general availability across both MongoDB Community Edition and Enterprise Server environments. To streamline infrastructure transitions, the newly introduced Application Modernization Platform leverages AI-powered tooling and specialized techniques to migrate legacy database systems two to three times faster.


### [MongoDB.local NYC 2025: AI 시대를 위한 이상적인 데이터베이스 정의](https://yomu.fyi/post/mongodb-local-nyc-2025-ai.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Dev Ittycheria, President and CEO, MongoDB
- Published: Sep 18, 2025
- Type: Announcement

MongoDB presented multiple updates at MongoDB.local NYC 2025 aimed at supporting modern AI workloads and enterprise modernization. The event detailed MongoDB 8.2 alongside Voyage AI embedding models and rerankers designed to enhance accuracy and efficiency in AI applications. Search and vector search capabilities have expanded into public preview for MongoDB Community Edition and Enterprise Server. Furthermore, the company launched the MongoDB Application Modernization Platform, which combines AI-driven tools and expert workflows to migrate legacy systems to MongoDB two to three times faster. MongoDB positions its JSON document model as a foundation providing memory, facts, and state continuity for agentic AI applications.


### [MongoDB.local NYC 2025: Definire il database ideale per l'era dell'AI](https://yomu.fyi/post/mongodb-local-nyc-2025-definire-il-database-ideale-per-l-era-dell-ai.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Dev Ittycheria, President and CEO, MongoDB
- Published: Sep 18, 2025
- Type: Announcement

MongoDB announced several product releases and platform updates aimed at supporting artificial intelligence and agentic workflows at MongoDB.local NYC. The release of MongoDB 8.2 arrives alongside integrations with Voyage AI embedding and reranker models designed to enhance data retrieval precision. In addition, MongoDB launched Search and Vector Search in public preview for both Community Edition and Enterprise Server deployments, bringing vector capabilities to self-managed environments. To assist organizations transitioning away from rigid legacy database systems, MongoDB also introduced the Application Modernization Platform, which combines AI-driven tooling and specialized migration expertise. Early benchmarks from the modernization platform demonstrate legacy migrations running two to three times faster while accelerating code rewriting tasks by an order of magnitude.


### [Real-time vs batch reconciliation: Practical patterns for keeping data in sync](https://yomu.fyi/post/real-time-vs-batch-reconciliation-practical-patterns-for-keeping-data.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: James Beswick
- Published: Aug 23, 2025
- Type: Explainer

Product data reconciliation across internal databases, ERPs, and platforms such as Stripe becomes harder to maintain as catalogs and system relationships grow, affecting financial reporting, customer experience, and operations. The post compares real-time reconciliation, using tools such as Kafka, Flink, Delta Live Tables, and Debezium for more immediate consistency, with batch approaches built around Spark, AWS Glue, Snowflake, and dbt for periodic, large-scale checks at lower cost and complexity. It also presents validation options including Great Expectations, Apache NiFi, Talend, and Airbyte, and recommends defining data ownership, applying field-specific conflict rules, and routing critical mismatches to manual review. Soft deletes and tombstones preserve deletion history, while observability through synchronization latency, conflict rates, resolution patterns, and regular audits helps assess reconciliation health as product catalogs scale.


### [From naive webhooks to durable sync: Queue-based and event-driven data reconciliation patterns](https://yomu.fyi/post/from-naive-webhooks-to-durable-sync-queue-based-and-event-driven-data.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: James Beswick
- Published: Aug 22, 2025
- Type: Explainer

Maintaining consistent product data between an internal catalog and Stripe becomes difficult when outages, throttling, high volumes, and changes from both systems create reconciliation failures. The post compares a naive one-way flow, in which a Lambda calls the Stripe API with no persistent synchronization record, with a queue-based design that durably stores changes, controls processing rate, retries failures with exponential backoff, and sends persistent failures to dead-letter queues. For bidirectional synchronization, it contrasts periodic ETL, using extraction, transformation, and loading, with an event-driven model based on Stripe Event Destinations, EventBridge, SQS, and symmetrical events from internal systems. The conclusion is that durable, observable queues and event-driven processing provide buffering, auditability, near-real-time propagation, independent change processing, and more graceful scaling for distributed reconciliation.


### [Keeping product data aligned with Stripe as your systems scale](https://yomu.fyi/post/keeping-product-data-aligned-with-stripe-as-your-systems-scale.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: James Beswick
- Published: Aug 21, 2025
- Type: Explainer

Keeping internal product data aligned with Stripe becomes an architectural challenge as catalogs grow from hundreds to millions of items. The problem spans independently operated systems with different data models, API constraints, rate limits, availability guarantees, and consistency models, while products may change faster than periodic reconciliation can complete. Stripe's payment-focused model covers identifiers, descriptions, prices, currencies, billing terms, and subscription behavior, whereas internal systems may contain categories, inventory, rich content, technical specifications, media, and sensitive cost data. The series examines manual verification, simple scripts, and periodic jobs for smaller catalogs, then points to streaming, change data capture, intelligent partitioning, and conflict resolution for high-scale reconciliation. It concludes that cross-system synchronization requires deliberate mapping and automated architecture because perfect synchronization is difficult to guarantee in eventually consistent distributed systems.


### [Forward Deployed Engineering](https://yomu.fyi/post/forward-deployed-engineering.md)
- Company: [Ramp](https://yomu.fyi/company/ramp.md)
- Author: Leo Mehr
- Published: Aug 5, 2025
- Type: Explainer

Forward Deployed Engineering (FDE) is presented as a customer-facing engineering function for B2B companies that need to integrate software deeply, customize implementations, and deliver outcomes for enterprise customers. Palantir is described as creating the role after its founding in 2003, with engineers deployed at military sites and customer offices; the model later spread to companies including Scale AI, C3.ai, Databricks, and Snowflake. At Ramp, the team grew from two to 16 FDEs after emerging in fall 2023, moving from firefighting toward generalized features, platforms, and AI that support enterprise product-market fit and new verticals. Its operating model spans prospects, implementation, rollout, and long-tail support, while “always be scoping” encourages direct requirement questioning, workarounds, and generalized solutions. The hiring guidance prioritizes drive, ownership, customer empathy, and communication alongside sufficient technical fundamentals, arguing that judgment about quick fixes versus scalable solutions is central to the role.


### [FlashList v2: A ground-up rewrite for React Native's New Architecture](https://yomu.fyi/post/flashlist-v2-a-ground-up-rewrite-for-react-native-s-new-architecture.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 31, 2025
- Type: Announcement

FlashList v2 is a ground-up rewrite of the React Native list component, aimed at removing v1’s dependence on item-size estimates, native layout corrections, and imprecise scrolling. Built for React Native’s New Architecture, it uses synchronous layout measurements to correct positions before paint and eliminates the need for native code. Its algorithm combines progressive rendering, predictions updated by item type, and rapid corrections, while adaptive rendering responds to scroll velocity, direction, and device performance. The new scrolling system progressively refines target positions for precise scrollToIndex operations, and horizontal lists can contain resizable items and coordinate with parent lists. The release also adds scroll-position maintenance, improved masonry support, recycle-pool controls, helper hooks, and better web support; benchmarks report up to 50% less blank area while scrolling than v1 on the new architecture.


### [Implementing scalable metered billing with Stripe: How Edgee handles billions of events](https://yomu.fyi/post/implementing-scalable-metered-billing-with-stripe-how-edgee-handles-bi.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Alex Casalboni
- Published: Jun 24, 2025
- Type: Problem & solution

Edgee needed usage-based billing for a business combining flat-rate subscriptions with charges for proxy requests and delivered data events, while processing billions of monthly requests across more than 100 edge locations. Its architecture captures web data at the edge through a proxy and records raw metering data centrally in Google BigQuery. To control several terabytes of data, Edgee aggregates usage by tenant and sends hourly Stripe meter events through AWS Lambda, with graduated pricing and idempotency handled by Stripe. A Go configuration defines meters, prices, flat-rate plans, and pricing tiers, while retry logic and idempotent updates support resilience during disruptions. The resulting billing system remains stateless and low-latency while providing accurate monthly invoicing and a simple hosted customer interface.


### [Measuring Commercial Impact at Scale at Canva](https://yomu.fyi/post/measuring-commercial-impact-at-scale-at-canva.md)
- Company: [Canva](https://yomu.fyi/company/canva.md)
- Author: Jun Ye
- Published: Jun 20, 2025
- Type: Problem & solution

Canva needed a scalable, standardized way to calculate the commercial impact of thousands of annual experiments on key business metrics like Monthly Active Users and Annual Recurring Revenue. Previously, disparate teams spent over six hours per experiment performing manual, error-prone calculations across fragmented spreadsheets and inconsistent data models. To resolve this, Canva built the IMPACT app using Snowflake, Streamlit, Snowpark, and Cortex to provide a self-serve platform tied directly to its central finance model. The application scales local uplift by actual audience exposure and supports pre-experiment scenario modeling alongside post-experiment tracking. A custom deployment workflow generates pull-request-isolated Streamlit environments in Snowflake stages, reducing time-to-insight to under ten minutes while enabling multiple developers to safely build and demo features in parallel.


### [Introducing the commerce payments protocol](https://yomu.fyi/post/introducing-the-commerce-payments-protocol.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 12, 2025
- Type: Announcement

Coinbase and Shopify are launching the Commerce Payments Protocol to support real-world commerce onchain, where direct token transfers do not handle checkout, settlement, fulfillment, cancellations, and refunds. The protocol uses an escrow smart contract to authorize payments by moving funds from the payer into escrow, then capture funds for the receiver later or in multiple partial increments. An operator role submits transactions for the parties, while payer-signed payment details constrain fields such as the receiver, token, maximum amount, and expiries. The protocol is available on Base and powers Shopify’s new USDC payment method, which is rolling out to millions of Shopify Payments stores; its six core operations include authorize, charge, void, reclaim, and refund, with the source also describing capture.


### [Evolution of Product Classification at Shopify: From Categories to Comprehensive Product Understanding](https://yomu.fyi/post/evolution-of-product-classification-at-shopify-from-categories-to-comp.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: May 8, 2025
- Type: Explainer

Shopify’s product classification system evolved from a 2018 logistic-regression model using TF-IDF into a product-understanding platform designed to capture categories, attributes, characteristics, metadata, and content-safety signals. A 2020 multimodal image-and-text approach improved classification, but growing product complexity led Shopify to combine Vision Language Models with its Standard Product Taxonomy, spanning more than 10,000 categories and over 1,000 attributes across more than 26 business verticals. The production pipeline uses two-stage category and attribute prediction, with category context shaping the second call, and runs on Kubernetes with NVIDIA GPUs; FP8 quantization, Nvidia Dynamo in-flight batching, and KV caching improve inference efficiency. A multi-LLM annotation system independently evaluates products and uses arbitration when models disagree, helping build training data for consistent predictions. The resulting system processes over 30 million predictions daily, delivers an 85% predicted-category acceptance rate, and doubles hierarchical precision and recall versus the earlier neural network approach.


### [Introducing Stripe Workflows: Tailoring Payments to Your Business Needs](https://yomu.fyi/post/introducing-stripe-workflows-tailoring-payments-to-your-business-needs.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Ben Smith
- Published: May 7, 2025
- Type: Announcement

Stripe introduces Stripe Workflows, a Dashboard-based system for orchestrating Stripe products and APIs through responsive, end-to-end automation. It addresses the complexity of coordinating multiple Stripe products by replacing custom event-handling and dependency-management code with visual workflows that support real-time triggers, branching logic, dynamic fields, and more than 600 event triggers. Workflows includes execution monitoring with event-payload and step-level tracing, 30-day default log retention, exponential-backoff retries, built-in idempotency, and recursion protection capped at a depth of five. The post describes applications including reducing chargeback risk, routing high-value transactions for compliance review, and creating custom-priced subscriptions from finalized quotes without additional infrastructure.


### [Behind the Scenes - A Glimpse into Tax Calculations](https://yomu.fyi/post/behind-the-scenes-a-glimpse-into-tax-calculations.md)
- Company: [Etsy](https://yomu.fyi/company/etsy.md)
- Author: Nancy Arnold
- Published: Mar 29, 2025
- Type: Explainer

As jurisdictions increasingly require marketplaces to collect and remit tax, Etsy describes the infrastructure it uses to calculate, record, and report taxes on transactions. When a buyer adds an item to a cart, Etsy sends a Quotation Request API call to Vertex, using buyer and seller locations, product category, tax rules, and mappings to determine the amount shown. At checkout and throughout fulfillment, DistributeTaxRequest SOAP calls carry order, cancellation, and refund details to Vertex, whose records support reporting and audit workflows imported into Etsy’s bigdata. The integration also required mapping Etsy taxonomy categories to Vertex rule drivers and separating quotation, distribution, and reporting workloads across multiple instances for horizontal scaling and load balancing. Shadow requests, ramped gradually without applying calculated taxes to carts, let the team simulate added Vertex load and monitor checkout performance before releasing traffic-increasing features.


### [Scaling Nextdoor’s Datastores: Part 5](https://yomu.fyi/post/scaling-nextdoor-s-datastores-part-5.md)
- Company: [Nextdoor](https://yomu.fyi/company/nextdoor.md)
- Author: Slava Markeyev
- Published: Mar 19, 2025
- Type: Problem & solution

Nextdoor addressed database and cache consistency issues caused by missed cache writes and concurrent read-fill operations in their look-aside architecture. While forward row versioning prevents out-of-order write inconsistencies, writer failures and race conditions during cache misses can leave stale data persisted in Redis. To resolve this, Nextdoor built a reconciliation pipeline that consumes PostgreSQL WAL replication logs with pg-bifrost, streams changes through Apache Kafka, and executes conditional deletions in Redis. The Go-based reconciler operates in two passes using a time wheel, running one pass in near real time and a second pass after a delay exceeding web request timeouts. Because conditional deletion evaluates row versions directly in the cache, the system processes change streams out of order and scales horizontally.


### [Scaling Nextdoor’s Datastores: Part 3](https://yomu.fyi/post/scaling-nextdoor-s-datastores-part-3.md)
- Company: [Nextdoor](https://yomu.fyi/company/nextdoor.md)
- Author: Ronak Shah
- Published: Mar 19, 2025
- Type: Problem & solution

Look-aside caching with object byte serialization can cause critical compatibility failures when application versions, dependencies, or database schemas change. Serialized cache entries bound to specific runtimes risk deserialization errors during deployments, triggering thundering herd queries against the underlying datastore. To resolve this, Nextdoor replaced runtime-bound serialization like Python pickle with MessagePack to serialize Django model objects. The team achieved forward compatibility by letting MessagePack ignore unrecognized new fields in older application code, while backward compatibility relies on mandatory default values for newly added model attributes. Nextdoor prepends a ten-byte header containing format metadata and version information before writing the serialized payload to cache stores.


### [Scaling Nextdoor’s Datastores: Part 2](https://yomu.fyi/post/scaling-nextdoor-s-datastores-part-2.md)
- Company: [Nextdoor](https://yomu.fyi/company/nextdoor.md)
- Author: Tushar Singla
- Published: Mar 19, 2025
- Type: Problem & solution

Nextdoor encountered scaling issues after adding read replicas when product engineers were initially tasked with deciding whether to route queries to the primary or replica databases. As business logic grew and gained abstraction layers, engineers struggled to track read-after-write consistency constraints across the call stack. To avoid replication lag race conditions, engineers routinely wrapped logic in database transactions, unintentionally directing all queries to the primary node and eroding read replica benefits over several years. The Core-Services team resolved this by injecting custom tracking logic into their Django ORM layer to monitor table writes during web requests and automate routing. They further optimized the system using a timing-based approach that restored replica read eligibility after the p99.9 replication lag elapsed.


### [Scaling Nextdoor’s Datastores: Part 1](https://yomu.fyi/post/scaling-nextdoor-s-datastores-part-1.md)
- Company: [Nextdoor](https://yomu.fyi/company/nextdoor.md)
- Author: Slava Markeyev
- Published: Mar 19, 2025
- Type: Explainer

Nextdoor relies on a Django backend connected to monolithic PostgreSQL databases and Redis look-aside caches. Migrating to distributed SQL datastores proved impractical because legacy business logic depends heavily on multi-table joins that could not be rewritten. Intermediate mitigations, including read replicas and data partitioning by severing foreign keys, extended infrastructure runways but left primary databases vulnerable to load bottlenecks. Furthermore, look-aside caching and replica usage introduced data staleness risks, atomicity losses across partitioned databases, and inconsistent cache-population behaviors. To address these limitations, Nextdoor initiated an architecture redesign focused on dynamic query routing to replicas, replica-driven cache hydration, time-bounded eventual consistency, and schema-resilient cache serialization.


[Newer posts](https://yomu.fyi/topic/scalability/page/11.md) · [Older posts](https://yomu.fyi/topic/scalability/page/13.md)