Loading…
Topic
375 posts about Scalability. Every summary links to the original.
See the idea
Consolidating agent memory, retrieval, and governance onto MongoDB Atlas removes sync and vendor layers, but ties scaling and control to one platform. This trade-off view shows the payoff, the cost, and the condition that decides it.
Autoscaling trades responsiveness, stability, and cost: this explainer shows how signal choice, scaling direction, and capacity limits shape the right control loop for a workload.
Apache Pinot is a distributed real-time OLAP datastore for low-latency analytics. This architecture view shows how streams, processing, partitioning, indexed storage, and replica-aware serving fit together.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
MongoDB announced several product releases and platform updates aimed at supporting artificial intelligence and agentic workflows at MongoDB.local NYC. The release of MongoDB 8.2 arrives alongside integrations with Voyage AI embedding and reranker models designed to enhance data retrieval precision. In addition, MongoDB launched Search and Vector Search in public preview for both Community Edition and Enterprise Server deployments, bringing vector capabilities to self-managed environments. To assist organizations transitioning away from rigid legacy database systems, MongoDB also introduced the Application Modernization Platform, which combines AI-driven tooling and specialized migration expertise. Early benchmarks from the modernization platform demonstrate legacy migrations running two to three times faster while accelerating code rewriting tasks by an order of magnitude.
Product data reconciliation across internal databases, ERPs, and platforms such as Stripe becomes harder to maintain as catalogs and system relationships grow, affecting financial reporting, customer experience, and operations. The post compares real-time reconciliation, using tools such as Kafka, Flink, Delta Live Tables, and Debezium for more immediate consistency, with batch approaches built around Spark, AWS Glue, Snowflake, and dbt for periodic, large-scale checks at lower cost and complexity. It also presents validation options including Great Expectations, Apache NiFi, Talend, and Airbyte, and recommends defining data ownership, applying field-specific conflict rules, and routing critical mismatches to manual review. Soft deletes and tombstones preserve deletion history, while observability through synchronization latency, conflict rates, resolution patterns, and regular audits helps assess reconciliation health as product catalogs scale.
Maintaining consistent product data between an internal catalog and Stripe becomes difficult when outages, throttling, high volumes, and changes from both systems create reconciliation failures. The post compares a naive one-way flow, in which a Lambda calls the Stripe API with no persistent synchronization record, with a queue-based design that durably stores changes, controls processing rate, retries failures with exponential backoff, and sends persistent failures to dead-letter queues. For bidirectional synchronization, it contrasts periodic ETL, using extraction, transformation, and loading, with an event-driven model based on Stripe Event Destinations, EventBridge, SQS, and symmetrical events from internal systems. The conclusion is that durable, observable queues and event-driven processing provide buffering, auditability, near-real-time propagation, independent change processing, and more graceful scaling for distributed reconciliation.
Keeping internal product data aligned with Stripe becomes an architectural challenge as catalogs grow from hundreds to millions of items. The problem spans independently operated systems with different data models, API constraints, rate limits, availability guarantees, and consistency models, while products may change faster than periodic reconciliation can complete. Stripe's payment-focused model covers identifiers, descriptions, prices, currencies, billing terms, and subscription behavior, whereas internal systems may contain categories, inventory, rich content, technical specifications, media, and sensitive cost data. The series examines manual verification, simple scripts, and periodic jobs for smaller catalogs, then points to streaming, change data capture, intelligent partitioning, and conflict resolution for high-scale reconciliation. It concludes that cross-system synchronization requires deliberate mapping and automated architecture because perfect synchronization is difficult to guarantee in eventually consistent distributed systems.
Forward Deployed Engineering (FDE) is presented as a customer-facing engineering function for B2B companies that need to integrate software deeply, customize implementations, and deliver outcomes for enterprise customers. Palantir is described as creating the role after its founding in 2003, with engineers deployed at military sites and customer offices; the model later spread to companies including Scale AI, C3.ai, Databricks, and Snowflake. At Ramp, the team grew from two to 16 FDEs after emerging in fall 2023, moving from firefighting toward generalized features, platforms, and AI that support enterprise product-market fit and new verticals. Its operating model spans prospects, implementation, rollout, and long-tail support, while “always be scoping” encourages direct requirement questioning, workarounds, and generalized solutions. The hiring guidance prioritizes drive, ownership, customer empathy, and communication alongside sufficient technical fundamentals, arguing that judgment about quick fixes versus scalable solutions is central to the role.
FlashList v2 is a ground-up rewrite of the React Native list component, aimed at removing v1’s dependence on item-size estimates, native layout corrections, and imprecise scrolling. Built for React Native’s New Architecture, it uses synchronous layout measurements to correct positions before paint and eliminates the need for native code. Its algorithm combines progressive rendering, predictions updated by item type, and rapid corrections, while adaptive rendering responds to scroll velocity, direction, and device performance. The new scrolling system progressively refines target positions for precise scrollToIndex operations, and horizontal lists can contain resizable items and coordinate with parent lists. The release also adds scroll-position maintenance, improved masonry support, recycle-pool controls, helper hooks, and better web support; benchmarks report up to 50% less blank area while scrolling than v1 on the new architecture.
Edgee needed usage-based billing for a business combining flat-rate subscriptions with charges for proxy requests and delivered data events, while processing billions of monthly requests across more than 100 edge locations. Its architecture captures web data at the edge through a proxy and records raw metering data centrally in Google BigQuery. To control several terabytes of data, Edgee aggregates usage by tenant and sends hourly Stripe meter events through AWS Lambda, with graduated pricing and idempotency handled by Stripe. A Go configuration defines meters, prices, flat-rate plans, and pricing tiers, while retry logic and idempotent updates support resilience during disruptions. The resulting billing system remains stateless and low-latency while providing accurate monthly invoicing and a simple hosted customer interface.
Canva needed a scalable, standardized way to calculate the commercial impact of thousands of annual experiments on key business metrics like Monthly Active Users and Annual Recurring Revenue. Previously, disparate teams spent over six hours per experiment performing manual, error-prone calculations across fragmented spreadsheets and inconsistent data models. To resolve this, Canva built the IMPACT app using Snowflake, Streamlit, Snowpark, and Cortex to provide a self-serve platform tied directly to its central finance model. The application scales local uplift by actual audience exposure and supports pre-experiment scenario modeling alongside post-experiment tracking. A custom deployment workflow generates pull-request-isolated Streamlit environments in Snowflake stages, reducing time-to-insight to under ten minutes while enabling multiple developers to safely build and demo features in parallel.
Coinbase and Shopify are launching the Commerce Payments Protocol to support real-world commerce onchain, where direct token transfers do not handle checkout, settlement, fulfillment, cancellations, and refunds. The protocol uses an escrow smart contract to authorize payments by moving funds from the payer into escrow, then capture funds for the receiver later or in multiple partial increments. An operator role submits transactions for the parties, while payer-signed payment details constrain fields such as the receiver, token, maximum amount, and expiries. The protocol is available on Base and powers Shopify’s new USDC payment method, which is rolling out to millions of Shopify Payments stores; its six core operations include authorize, charge, void, reclaim, and refund, with the source also describing capture.
Shopify’s product classification system evolved from a 2018 logistic-regression model using TF-IDF into a product-understanding platform designed to capture categories, attributes, characteristics, metadata, and content-safety signals. A 2020 multimodal image-and-text approach improved classification, but growing product complexity led Shopify to combine Vision Language Models with its Standard Product Taxonomy, spanning more than 10,000 categories and over 1,000 attributes across more than 26 business verticals. The production pipeline uses two-stage category and attribute prediction, with category context shaping the second call, and runs on Kubernetes with NVIDIA GPUs; FP8 quantization, Nvidia Dynamo in-flight batching, and KV caching improve inference efficiency. A multi-LLM annotation system independently evaluates products and uses arbitration when models disagree, helping build training data for consistent predictions. The resulting system processes over 30 million predictions daily, delivers an 85% predicted-category acceptance rate, and doubles hierarchical precision and recall versus the earlier neural network approach.
Stripe introduces Stripe Workflows, a Dashboard-based system for orchestrating Stripe products and APIs through responsive, end-to-end automation. It addresses the complexity of coordinating multiple Stripe products by replacing custom event-handling and dependency-management code with visual workflows that support real-time triggers, branching logic, dynamic fields, and more than 600 event triggers. Workflows includes execution monitoring with event-payload and step-level tracing, 30-day default log retention, exponential-backoff retries, built-in idempotency, and recursion protection capped at a depth of five. The post describes applications including reducing chargeback risk, routing high-value transactions for compliance review, and creating custom-priced subscriptions from finalized quotes without additional infrastructure.
As jurisdictions increasingly require marketplaces to collect and remit tax, Etsy describes the infrastructure it uses to calculate, record, and report taxes on transactions. When a buyer adds an item to a cart, Etsy sends a Quotation Request API call to Vertex, using buyer and seller locations, product category, tax rules, and mappings to determine the amount shown. At checkout and throughout fulfillment, DistributeTaxRequest SOAP calls carry order, cancellation, and refund details to Vertex, whose records support reporting and audit workflows imported into Etsy’s bigdata. The integration also required mapping Etsy taxonomy categories to Vertex rule drivers and separating quotation, distribution, and reporting workloads across multiple instances for horizontal scaling and load balancing. Shadow requests, ramped gradually without applying calculated taxes to carts, let the team simulate added Vertex load and monitor checkout performance before releasing traffic-increasing features.
Nextdoor addressed database and cache consistency issues caused by missed cache writes and concurrent read-fill operations in their look-aside architecture. While forward row versioning prevents out-of-order write inconsistencies, writer failures and race conditions during cache misses can leave stale data persisted in Redis. To resolve this, Nextdoor built a reconciliation pipeline that consumes PostgreSQL WAL replication logs with pg-bifrost, streams changes through Apache Kafka, and executes conditional deletions in Redis. The Go-based reconciler operates in two passes using a time wheel, running one pass in near real time and a second pass after a delay exceeding web request timeouts. Because conditional deletion evaluates row versions directly in the cache, the system processes change streams out of order and scales horizontally.
Look-aside caching with object byte serialization can cause critical compatibility failures when application versions, dependencies, or database schemas change. Serialized cache entries bound to specific runtimes risk deserialization errors during deployments, triggering thundering herd queries against the underlying datastore. To resolve this, Nextdoor replaced runtime-bound serialization like Python pickle with MessagePack to serialize Django model objects. The team achieved forward compatibility by letting MessagePack ignore unrecognized new fields in older application code, while backward compatibility relies on mandatory default values for newly added model attributes. Nextdoor prepends a ten-byte header containing format metadata and version information before writing the serialized payload to cache stores.
Nextdoor encountered scaling issues after adding read replicas when product engineers were initially tasked with deciding whether to route queries to the primary or replica databases. As business logic grew and gained abstraction layers, engineers struggled to track read-after-write consistency constraints across the call stack. To avoid replication lag race conditions, engineers routinely wrapped logic in database transactions, unintentionally directing all queries to the primary node and eroding read replica benefits over several years. The Core-Services team resolved this by injecting custom tracking logic into their Django ORM layer to monitor table writes during web requests and automate routing. They further optimized the system using a timing-based approach that restored replica read eligibility after the p99.9 replication lag elapsed.
Nextdoor relies on a Django backend connected to monolithic PostgreSQL databases and Redis look-aside caches. Migrating to distributed SQL datastores proved impractical because legacy business logic depends heavily on multi-table joins that could not be rewritten. Intermediate mitigations, including read replicas and data partitioning by severing foreign keys, extended infrastructure runways but left primary databases vulnerable to load bottlenecks. Furthermore, look-aside caching and replica usage introduced data staleness risks, atomicity losses across partitioned databases, and inconsistent cache-population behaviors. To address these limitations, Nextdoor initiated an architecture redesign focused on dynamic query routing to replicas, replica-driven cache hydration, time-bounded eventual consistency, and schema-resilient cache serialization.
Payment processors such as Stripe emit numerous transaction events, while dashboard and API-based or batch approaches can limit historical analysis, real-time visibility, and correlation with business metrics as volumes grow. The proposed pipeline sends signed Stripe webhooks through API Gateway to Lambda for validation and enrichment, buffers them in Kinesis Data Streams, and uses a second Lambda to transform and index records in OpenSearch. OpenSearch Dashboards provides visualizations, while CloudWatch tracks pipeline health; Kinesis retention and replay support recovery and reprocessing. The post also describes a Kinesis Data Firehose route that can remove the consumer Lambda and reduce costs but may add batching latency. The resulting design is presented as scalable to millions of transactions per day, with sub-minute payment metrics visibility and flexible historical querying, while security guidance includes encryption, IAM configuration, and regular audits.
The guide describes a multi-region payment-processing architecture for applications that use Stripe’s API and need resilience against regional outages, latency, rate limits, and payment-data compliance constraints. It routes requests with Amazon Route 53 health checks and configurable latency-based, weighted, or geolocation policies, while identical regional API Gateway and Lambda deployments process payments. DynamoDB Global Tables replicate payment state and support distributed rate limiting, with optimistic locking, regional token-bucket allocation, retries, and exponential backoff for Stripe requests. The sample Lambda records a payment as PENDING, creates and confirms a Stripe PaymentIntent, then updates the state with the Stripe payment ID and returned status; the conclusion presents the design as a foundation for regional resilience, state consistency, and API-rate management.
Canva launched DesignDNA in December 2024 as a personalized year-in-review campaign to highlight user achievements and showcase generative AI capabilities. Because internal privacy rules strictly prohibited inspecting personal designs, the engineering and creative teams inferred user preferences from the style and theme metadata tags on public templates. An initial keyword-matching algorithm paired 95% of users with one of seven emerging design trends, while generative AI keyword expansion increased coverage to 99%. The team also synthesized over one million localized poems and distinct design personalities using tools like Magic Write and Dream Lab. Finally, dynamic Canva template elements populated through URL parameters assembled 95 million distinct multi-page stories across nine locales.
Reliable Stripe webhook processing must account for lost deliveries from network or service outages, out-of-order events, and duplicates caused by Stripe retries. The proposed AWS architecture uses API Gateway for signature validation and throttling, an SQS FIFO queue for ordered delivery and content-based deduplication, Lambda for processing, DynamoDB for event-ID idempotency, and an SQS DLQ for failed messages. In the CloudFormation example, the main queue has a 300-second visibility timeout and sends messages to the FIFO DLQ after three receives; DynamoDB records expire through a seven-day TTL. Lambda retries failures with exponential backoff, while CloudWatch monitors queue depth, latency, and errors. The design is presented as a scalable foundation, with multi-region failover available at added cost and complexity, though single-region deployment may suffice for many applications.
The best engineering reads, once a week.
